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1  Overview 


Active  networks  support  dynamic  generalization  of  network  services  by  injecting  code  and/or  pol¬ 
icy  into  the  shared  nodes  of  the  network.  Early  active  networking  projects  have  focused  on  plat¬ 
forms  and  enabling  technology  such  as  programming  languages  and  lower-level  architectural  fea- 
tures.These  projects  achieved  significant  and  interesting  results  in  terms  of  expressive  power,  flex¬ 
ibility  and  security.  Other  projects  applied  active  network  technology  to  solve  application-specific 
problems,  such  as  network  management,  interest-filtering  and  time  management  in  distributed  sim¬ 
ulation,  and  video  streaming.  The  Activecast  project  was  initiated  to  help  bridge  the  gap  between 
these  two  extremes.  To  improve  the  useability  of  active  networks,  “application-friendly”  APIs  for 
high-level  customizable  services  were  needed.  These  services  would  be  designed  to  hide  the  com¬ 
plexity  of  the  programming  process  while  allowing  applications  to  conveniently  specify  the  pro¬ 
cessing/handling  they  would  like  their  data  to  receive. 

This  report  describes  the  technical  results  of  the  Activecast  project:  PAMcast,  a  programmable 
form  of  anycast  that  uses  application-specific  criteria  to  select  one  (or  more)  destinations  from  a 
set  of  possible  destinations;  Concast  (and  Secure  Concast),  a  many-to-one  channel  that  can  be  pro¬ 
grammed  to  merge  messages  as  they  travel  through  the  network  from  multiple  senders  toward  a 
single  receiver;  Ephemeral  State  Processing  (ESP),  an  ultra-lightweight  form  of  active  network¬ 
ing  designed  to  support  auxiliary  computations  in  the  network;  LightWeight  Processing  modules 
(LWP),  a  paradigm  for  deployment  of  per-user  services  in  the  network;  and  Speccast,  a  generalized 
routing  and  forwarding  service  in  which  addresses  are  replaced  by  predicates  that  specify  packet 
destinations. 

These  services  represent  a  middle  point  between  general-purpose,  application-independent  plat¬ 
forms  (offered  by  Execution  Environments  (EEs)  and  Node  Operating  Systems)  and  application- 
specific  solutions  (such  as  reliable  data  distribution).  .  They  are  designed  to  provide  high-level 
customizable  services  that  applications,  particularly  large  group  applications,  can  use  to  perform 
topology- sensitive  processing  without  knowing  the  details  of  the  network.  The  services  hide  the 
complexity  of  active  networks  and  achieve  scalability  via  features  such  as  anonymity  (e.g.,  pro¬ 
grammable  communication  with  unknown  end-systems  over  unknown  network  routers),  automatic 
code  deployment  (to  only  “the  right  nodes”),  simplified  router  state  management,  and  enforcable 
security  and  admission  control.  We  argue  that  they  are  scalable,  both  in  the  sense  of  supporting 
applications  involving  many  endpoints,  and  in  the  sense  of  supporting  widespread  deployment  and 
simultaneous  usage  by  many  applications.  Concast,  PAMcast,  LWP  and  Speccast  are  EE-agnostic — 
they  could  be  implemented  on  any  of  the  currently  extant  EE  platforms — while  ESP  should  be  re¬ 
garded  as  a  very  lightweight  EE,  which  is  designed  to  be  implemented  mainly  in  special  hardware. 
These  services  are  also  generally  backward-compatible,  in  that  they  could  be  deployed  incremen¬ 
tally  in  the  Internet,  given  appropriate  (specialized)  router  support. 

This  report  is  organized  as  follows.  The  next  section  outlines  some  lessons  learned  from  the 
project,  including  key  principles  for  designing  scalable,  usable  services.  We  then  describe  the  PAM¬ 
cast,  Concast  and  Secure  Concast,  ESP,  LWP,  and  Speccast  services,  highlighting  their  embodiment 
of  those  principles  and  giving  example  uses  of  the  services.  We  then  briefly  describe  other  tech¬ 
nology  developed  in  the  project,  including  some  that  emerged  as  a  by-product,  and  some  whose 
development  is  still  ongoing.  In  the  final  section,  tangible  outputs  of  the  project  are  tallied. 
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2  Designing  for  Application-Friendliness 

To  bridge  the  gap  between  general-purpose  programmable  platforms  and  applieation-speeifie  solu¬ 
tions,  aetive  networks  should  support  application-friendly  serviees:  high-level  serviees  that  improve 
the  usability  and  scalability  of  aetive  networks. 

Usability  refers  to  the  amount  of  effort  required  by  the  applieation  programmer  to  make  use  of 
the  serviee.  It  seems  elear  that  end  users  are  no  more  likely  to  program  the  network  than  they  are 
to  program  their  home  eomputers.  To  be  attraetive,  aetive  serviees  should  expose  only  the  details 
that  are  relevant  to  the  application.  For  example,  serviees  that  require  the  applieation  programmer 
to  explieitly  identify  many  nodes,  partieipants,  or  ehannels,  or  to  know  the  speeifie  topology  of  any 
signifieant  part  of  the  network,  are  inherently  difficult  to  program  and  do  not  scale. 

Scalability  is  important  in  two  ways.  First,  an  aetive  serviee  should  support  applieations  involv¬ 
ing  large  numbers  of  end  systems  and  network  nodes.  It  is  widely  reeognized  that  large-seale  group 
applieations  are  impraetieal  without  some  form  of  network  support.  Although  the  aetive  eode  needs 
to  be  deployed  and  run  on  a  large  number  of  routers  and  end-systems,  the  applieation  should  not  be 
required  to  know  how  many  or  whieh  nodes  are  involved  in  the  eomputation.  Seeond,  the  system 
should  seale  in  terms  of  the  number  of  simultaneous  aetive  serviees  that  ean  be  supported.  Beeause 
aetive  serviees  eonsume  router  state  and  proeessing  eyeles,  it  is  important  that  the  serviee  regu- 
late/poliee  the  resourees  eonsumed  by  the  applieation  using  the  aetive  serviee  so  that  the  sealability 
is  maximized  and  denial-of-serviee  (either  intentionally  or  aeeidentally)  is  prevented.  Resouree 
management  and  admission  eontrol  add  eomplexity  in  the  form  of  polieies  governing  whieh  users 
are  allowed  to  invoke  whieh  serviees.  The  admission  eontrol  meehanism  itself  must  have  resouree 
bounds,  lest  a  denial-of-serviee  attaek  be  mounted  by  saturating  the  admission  eontrol  meehanism. 

The  following  highlights  eharaeteristies  of  serviees  that  we  believe  are  neeessary  for  usability 
and  sealability: 

Anonymity:  Anonymity  improves  sealability  both  in  terms  of  network  state  maintained  and  ease 
of  programming.  Serviees  that  distinguish  among  users  (or  paekets/nodes)  require  (explieit  or 
implieit)  state  and/or  policies  that  define  how  eaeh  user  (or  paeket/node)  is  treated.  Serviees 
that  deal  with  “groups”  of  users/paekets/nodes  are  also  easier  to  program.  For  example,  it  is 
more  eonvenient  to  transmit  a  single  IP  multieast  paeket  than  N  unieast  paekets.  Similarly, 
treating  all  paekets  in  a  elass  or  group  the  same  reduees  network  state  and  paeket  proeessing 
overhead,  whieh  is  important  if  nodes  must  proeess  paekets  at  line  speed. 

Locality  and  Automatic  Deployment:  Deploying  flow-speeihe  state  and/or  proeessing  at  every 
node  along  a  flow’s  path  does  not  seale  well.  Flow  state  and  proeessing  should  only  be 
earried  out  where  needed.  Serviees  that  automatieally  determine  nodes  where  eode  is  needed 
and  deploy  eode  at  those  points  have  better  usability  and  sealability  than  serviees  that  require 
the  applieation  to  do  this.  Moreover,  determination  of  whether  proeessing  is  needed  at  a  node 
should  be  done  periodieally,  not  per-paeket. 

Specialization:  A  narrow  but  customizable  programming  interface  is  ideal.  Application-designers 
do  not  want  to  deal  with  complex  distributed  algorithms.  The  service  should  provide  a  re¬ 
stricted  interface,  with  support  functionality  (e.g.  code  deployment)  hidden  from  the  user. 

Automatic  State  Management:  Any  “interesting”  active  network  service  requires  network  state. 
Requiring  end-systems  to  manage/police  this  state  is  not  desirable  or  scalable.  State  should 
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be  set  up,  policed,  and  destroyed  by  the  service,  not  the  application. 

Security  and  Authentication:  Complex  trust  models  in  which  packets,  intermediate  nodes,  etc 
must  be  trusted  and  authenticated  are  difficult  to  make  secure.  Services  that  use  simple  trust 
models  (e.g.,  only  trusting  end-systems)  are  easier  to  implement  and  require  less  overhead. 

Best  Effort:  Best-effort  services  may  place  additional  burdens  on  the  end  systems,  forcing  them 
to  recover  from  packet  loss.  However,  such  services  exhibit  better  scalability  and  soft-state 
techniques  can  be  used  to  help  manage  network  state. 

The  following  sections  present  four  active  network  services  designed  to  ease  the  task  of  pro¬ 
gramming  active  networks  (usability)  and  improve  scalability. 

3  Programmable  Any-Multicast 

PAMcast  is  based  on  the  traditional  anycast  service  specified  by  Parfridge,  Mendez,  and  Milliken 
in  RFC  1546,  in  which  fhe  nefwork  delivers  a  packef  from  a  sender  fo  any  of  a  sef  of  receivers. 
The  realizafion  of  such  a  service  has  been  considered  af  fhe  nefwork  layer,  where  fhe  receiver  is 
selected  based  on  closesl  hop  counf  or  closesl  Autonomous  System  (AS)  counf.  Using  hop  counf  or 
AS  counf  as  fhe  selection  criferia  aims  fo  reduce  nefwork  resource  usage.  However  if  is  clear  fhaf 
hop-counf-orienfed  selecfion  is  limited  and  does  nof  meef  fhe  needs  of  diverse  applicafions. 

One  can  envision  af  teas!  fwo  ways  fo  generalize  fhe  fradifional  anycasf  service.  Firsf,  one  mighf 
provide  more  flexibilily  in  fhe  specificafion  of  how  fhe  receiver  is  selecfed.  Several  projecfs  have 
pursued  fhis  direction  using  application-layer  realizafions  of  anycasting,  for  example  Berkeley’s 
Shared  Passive  Nefwork  Performance  Discovery  (SPAND)  and  Georgia  Tech’s  Applicafion-level 
Anycasfing  Service.  Second,  one  mighf  provide  more  flexibilify  in  fhe  number  of  receivers  fhaf  are 
selected,  allowing  more  fhan  jusf  one  to  receive  a  packef.  If  fhe  sef  of  receivers  has  size  n,  selecfing 
any  one  member  corresponds  fo  fradifional  anycasf  service,  white  selecfing  any  n  corresponds  fo 
fradifional  mulficasf  service.  Values  befween  1  and  n  represenf  a  new  form  of  service  (sometimes 
called  partial  mulficasf). 

We  are  exploring  bofh  forms  of  generalizafion,  fargeffing  a  service  fhaf  allows  flexibilily  in  bolh 
fhe  number  and  melhod  for  selecting  receivers  from  a  sef.  A  more  complefe  descripfion  of  our  work 
can  be  found  in  [6]. 

3.1  PAMcast  Service  Overview  and  Examples 

We  propose  a  new  packef  delivery  service  —  programmable  any-mulficasf  or  PAMcasf  —  which 
generalizes  bolh  anycasf  and  mullicasl  services,  by  providing  for  delivery  to  any  m  oul  of  n  group 
members,  1  <  m  <  n.  Such  a  service  has  applicabilily  for  a  wide  range  of  applicafions.  For 
example: 

•  Fault  tolerant  repositories.  Consider  fhe  problem  of  storing  a  data  item  in  a  repository. 
For  fault  tolerance,  the  repository  service  offers  a  number  of  geographically  diverse  storage 
locations.  One  might  PAMcast  the  data  item  to  m  of  the  n  storage  locations,  with  m  selected 
based  on  the  importance  of  the  data,  the  cost  of  storage  in  multiple  locations,  and  the  perceived 
reliability  of  the  network  and  storage  servers.  The  choice  of  the  particular  m  locations  might 
attempt  to  balance  the  load  over  time. 


3 


•  Parallel  cache  queries.  Suppose  a  group  of  caches  store  data  items.  A  client  might  PAMcast 
a  query  to  m  caches  in  the  group,  in  the  hope  that  at  least  one  has  the  desired  item.  The 
choice  of  m  must  balance  the  penalty  associated  with  not  finding  the  item  with  the  overhead 
associated  with  query  to  and  delivery  from  multiple  caches.  It  should  also  take  into  account 
the  probability  that  each  cache  has  the  desired  item;  when  an  item  is  more  densely  replicated, 
a  smaller  value  for  m  can  be  used  [5]. 

•  Parallel  download.  Suppose  multiple  servers  have  the  same  content.  A  client  might  PAM¬ 
cast  queries  to  m  of  the  servers,  requesting  that  each  transmit  a  portion  of  a  particular  file.  If 
is  well-known  fhaf  parallel  downloads  improve  clienf  response  lime  over  single-server  down¬ 
loads.  The  value  of  m  mighl  depend  on  fhe  file  size;  fhe  parlicular  m  servers  mighf  be  chosen 
lo  be  relatively  close  lo  fhe  clienf. 

As  fhe  sample  applicafions  indicate,  some  additional  confrol  over  fhe  delivery  (beyond  fhe  size 
m)  is  desirable.  Mosf  generally,  one  can  envision  fhaf  each  packef  confains  a  program  (or  reference 
lo  a  program)  fhaf  confrols  how  fhe  m  receivers  are  selected.  We  consider  fhe  implemenlalion  of 
several  specific  modes  of  delivery,  lo  explore  whal  is  possible  wilh  limiled  slafe  and  compufafion 
al  fhe  roulers.  Thus,  fhe  PAMcasl  service  is  programmable  in  Iwo  dimensions:  fhe  number  m  of 
receivers  and  fhe  mode  of  seleclion. 

3.2  PAMcast  Architecture 

A  naive  implemenlalion  of  fhe  PAMcasl  service  would  Iransmil  a  packel  lo  all  n  group  members  (as 
in  mullicasl),  Ihen  filler  al  Ihe  receivers  lo  deliver  lo  m  members.  Such  an  approach,  however,  makes 
poor  use  of  nelwork  resources  when  m  «  n.  If  also  musl  address  Ihe  problem  of  determining 
fillers  lhal  selecl  m.  An  alternative  approach  is  lo  conslrucl  independenl  mullicasl  groups,  one  for 
each  value  of  m.  However,  Ihe  number  of  groups  is  potentially  very  large  (2”  if  all  possible  groups 
of  all  sizes  are  supported). 

We  design  an  archileclure  for  scalable  realization  of  Ihe  service  using  selective  copying  al  branch 
poinls  in  a  free  lhal  includes  all  n  receivers.  The  archileclure  can  be  implemented  eilher  wilhin  an 
application-layer  overlay  or  wilhin  nelwork  routers,  Ihough  we  discuss  only  Ihe  router  implemenla- 
lion  in  Ibis  paper.  If  implemented  in  routers,  partial  deploymenl  is  feasible,  wilh  lunneling  belween 
capable  routers.  The  basic  service  is  best-effort  in  Ihe  sense  lhal  il  cannol  guarantee  delivery  lo 
exaclly  m  receivers. 

Our  archileclure  is  based  on  a  shared  free.  We  use  Ihe  term  “core”  lo  denote  Ihe  rool  of  Ihe 
shared  free.  The  Iree-based  archileclure  provides  a  scalable  means  lo  deliver  a  specified  number  of 
copies  of  a  message  lo  group  members.  Unlike  a  mullicasl  free,  Ihe  PAMcasl  free  has  an  additional 
allribule,  group  size,  mainlained  on  a  per  group  and  per  free  link  basis.  The  group  size  denotes  Ihe 
number  of  downslream  group  members  lhal  are  reachable  Ihrough  Ihe  link,  where  “downslream” 
means  “in  Ihe  direction  away  from  Ihe  rool”. 

Figure  1  shows  how  messages  are  delivered  Ihrough  a  PAMcasl  free.  PAMcasl  messages  have 
Ihree  header  fields:  group  address,  degree  and  mode.  The  degree  field  is  Ihe  only  one  shown  in 
Ihe  figure,  since  il  is  Ihe  only  one  modified  during  Iransmission.  The  group  address  field  indicates 
Ihe  largel  group  of  members  lo  which  Ihe  message  should  be  delivered.  The  degree  field  indicates 
Ihe  largel  number  of  group  members  lhal  are  supposed  lo  receive  Ihe  message.  The  mode  field 
indicates  how  Ihe  copies  are  made,  eilher  by  specifying  a  buill-in  melhod  or  providing  a  reference 
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Figure  1 :  Message  delivery  on  a  PAMcast  tree 


to  a  program.  End  applications  specify  the  three  attributes  on  per-message  basis  according  to  their 
specific  needs. 

When  an  application  sends  a  PAMcast  message,  the  message  is  first  routed  to  the  core  router 
of  the  group.  Upon  receiving  a  PAMcast  message,  the  core  router  identifies  which  delivery  mode 
is  used  and  whaf  fhe  degree  of  fhe  message  is.  Using  fhe  mode  and  fhe  degree  informafion,  fhe 
core  roufer  defermines  i)  a  subsef  of  incidenf  free  links  fhrough  which  duplicafed  messages  are  fo 
be  roufed  and  ii)  fhe  degree  of  each  duplicafed  message.  The  mode  of  fhe  message  confrols  how  fo 
selecf  fhe  subsef  of  fhe  free  links  and  fhe  disfribufion  of  degree  among  fhe  selecfed  free  links.  The 
above  operations  are  repealed  on  each  free  router  while  PAMcasf  messages  are  fraveling  along  fhe 
free. 

For  PAMcasf  free  managemenf,  we  propose  a  PAMcasf  Group  Membership  Prolocol  (PGMP), 
which  is  similar  fo  Infemef  Group  Membership  Prolocol  (IGMP).  PGMP  has  fhree  conlrol  messages, 
group  join,  group  leave,  and  group  refresh.  There  are  Iwo  main  differences  belween  IGMP  and 
PGMP  in  processing  fhe  conlrol  messages.  Firsl,  PGMP  messages  are  roufed  all  fhe  way  fo  fhe  core 
roufer.  In  IGMP,  however,  messages  are  ferminaled  al  fhe  firsl  roufer  lhal  is  pari  of  a  delivery  free  for 
fhe  group.  The  purpose  of  relaying  fhe  PGMP  messages  up  fo  fhe  core  roufer  is  fo  updale  fhe  group 
size  allribufes  on  every  free  links  along  fhe  pafh  from  fhe  joining  hosl  fo  fhe  core  roufer.  Second, 
PGMP  messages  conlain  a  group  size  field.  The  group  size  field  of  an  incoming  conlrol  message 
represenls  fhe  lolal  number  of  downslream  group  members  reachable  fhrough  fhe  incoming  free  link 
for  fhe  conlrol  message.  A  roufer  fhaf  receives  a  PGMP  message  will  use  fhe  group  size  field  fo  sel 
fhe  group  size  value  for  fhe  incoming  link.  Each  member  of  fhe  PAMcasf  group  will  periodically 
send  a  refresh  message  fo  reinslanliale  fhe  group  size  allribufe  along  fhe  pafh  fo  fhe  core. 
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3.3  PAMcast  Performance  Evaluation 

We  evaluate  the  performanee  of  two  speeifie  methods  for  seleeting  the  m  reeeivers: 

Balanced  Mode.  The  goal  of  this  mode  is  to  aehieve  equal  distribution  of  messages  over  the  set 
of  group  members.  The  rationales  for  the  balaneed  mode  are  i)  inereased  load-balaneing 
among  group  members,  ii)  inereased  fault-toleranee  to  link  or  node  failures  by  avoiding  a 
eoneentration  of  messages  on  group  members  that  share  the  same  tree  links/nodes,  and  iii) 
random  seleetion  of  representative  group  members  that  are  loeated  sparsely  on  the  underlying 
network. 

We  implement  the  balaneed  mode  using  a  weighted  eounter  (WC)  for  eaeh  tree  link,  whieh 
is  similar  to  the  virtual  eloek  for  weighted  fair  queueing.  The  weighted  eounter  keeps  traek 
of  the  total  degree  of  the  messages  passing  through  the  tree  link,  whieh  is  normalized  to  the 
number  of  downstream  group  members.  The  load-balaneing  using  the  weighted  eounters  is 
aeeomplished  by  balaneing  the  eounters  of  the  tree  links. 

Closest  Mode.  The  closest  mode  delivers  a  message  to  the  k  group  members  that  are  elosest  (in  hop 
eount)  to  the  eore  router.  The  underlying  rationale  of  the  elosest  mode  is  to  reduee  the  network 
lateney  and  the  total  network  bandwidth  usage  by  seleeting  the  m-elosest  group  members 
from  the  eore  router.  We  eonsider  two  different  approaehes  to  the  elosest  mode.  First,  a 
deterministie  elosest  meehanism  delivers  messages  to  preeisely  the  elosest  group  members. 
The  deterministie  meehanism  requires  that  a  router  should  maintain  per-member  distanee 
information  for  all  downstream  group  members.  This  ean  be  aehieved  by  adding  one  more 
field,  hop-distanee,  into  PGMP  messages.  Upon  reeeiving  a  PGMP  message,  a  router  eolleets 
the  distanee  information  of  the  sending  group  member,  inereases  the  hop-distanee  by  one 
and  passes  the  message  to  the  parent  router  as  usual.  The  state  required  by  the  deterministie 
meehanism  is  linear  in  the  number  of  downstream  group  members,  potentially  prohibitive  for 
large  and  eoneentrated  groups. 

Alternatively,  a  probabilistie  elosest  meehanism  delivers  messages  to  the  elosest  group  mem¬ 
bers  with  high  probability  (that  is,  with  high  probability  the  members  seleeted  are  the  elosest 
to  the  eore).  This  meehanism  uses  a  eonstant  amount  of  state  per  on-tree  router.  We  use  a  de¬ 
gree  distribution  meehanism  based  on  the  Chebyehev  inequality  for  the  probabilistie  elosest 
mode. 

To  analyze  performanee,  we  used  the  ns-2  simulation  paekage  to  model  the  PAMeast  serviee. 
For  our  study,  we  used  five  differenf  random  fopologies  of  100  nodes.  The  simulafion  model  eon- 
sisfs  of  fwo  separafe  modules:  a  dafa  forwarding  module  and  a  free  managemenf  module.  The  dafa 
forwarding  module  af  eaeh  roufer  handles  PAMeasf  messages  and  provides  appropriafe  degree  dis- 
fribufion  and  message  eopying/forwarding  depending  on  fhe  delivery  mode  of  messages.  The  free 
managemenf  module  implemenfs  fhe  PGMP  profoeol. 

Whaf  follows  are  some  sample  resulfs;  for  more  extensive  evaluafion,  see  [6].  Firsf,  Figure  2(a) 
shows  how  fhe  WC  algorifhm  works  when  new  members  join  a  PAMeasf  group.  Af  fhe  beginning, 
members  ml,  m2  and  m3  join  fhe  group.  During  0  <  f  <  3000,  fhe  fhree  members  reeeive  fhe  equal 
number  of  messages.  Att  —  3000,  fhree  new  members  m4,  m5  and  m6  join  fhe  group,  resulfing 
in  fhe  fofal  six  group  members.  During  3000  <  t  <  6000,  fhe  six  members  equally  reeeive  fhe 
messages.  WC  algorifhm  guaranfees  fhaf  fhe  new  fhree  group  members  reeeive  messages  wifh  fhe 
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(a)  Messages  received  as  members  join 


Time 


(b)  Messages  received  as  members  leave 


Figure  2:  Message  distribution  with  WC  algorithm 


same  rate  of  the  previously  joined  members.  Att  —  6000,  other  three  members  mV,  m8,  m9  join 
the  group.  During  t  >  6000,  all  the  nine  group  members  reeeive  the  same  number  of  messages.  In 
brief.  Figure  2(a)  shows  that  the  WC  algorithm  works  well  with  the  member  join  aetivities. 

Figure  2(b)  shows  how  the  WC  algorithm  works  when  members  leave  the  PAMeast  group.  The 
graph  shows  similar  performanee  with  the  member  join  ease.  At  the  beginning  nine  group  members 
join  the  group.  Att  —  3000,  three  members  m4,  m5  and  m6  leave  the  group,  resulting  in  the  total 
six  group  members.  During  3000  <  t  <  6000,  the  message  delivery  rate  at  eaeh  group  member 
inereases  aeeordingly,  eompared  to  that  of  f  <  3000.  Figures  2(a)  and  2(b)  have  shown  the  load- 
balaneing  property  of  the  WC  algorithm  is  well-maintained  with  group  join/leave  aetivities. 


Time 


Figure  3:  Probabilistie  elosest  with  m  =  4 
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Figure  3  shows  how  PAMcast  messages  are  delivered  in  the  elosest  mode.  The  total  group 
size  is  ten  and  the  target  degree  of  PAMeast  messages  is  four.  (In  addition,  Table  1  shows  the  hop 
distanee  distribution  of  group  members  on  the  PAMeast  tree  for  Figure  3.)  Sinee  the  target  degree 
is  four,  in  an  ideal  elosest  mode,  members  m4,  m7  and  m8  always  reeeive  messages  and  one  of  the 
hop-distanee  four  members,  m3,  m9  and  mlO,  would  reeeive  remaining  messages.  The  probabilistie 
elosest  always  delivers  messages  to  member  m4,  m7,  and  m8  as  depieted  in  Figure  3,  whieh  is  same 
with  the  ideal  elosest.  Among  the  hop-distanee  four  members,  preferenee  goes  in  the  order  of  m9, 
m3  and  mlO.  The  remaining  group  members  reeeive  no  messages  at  all. 

Table  1 :  Hop  distanee  of  group  members  in  a  tree  for  Figure3 


Hop  disfance 

Group  Members 

2 

m4  and  m7 

3 

m8 

4 

m3,  m9  and  mlO 

5 

ml,  m2,  m5  and  m6 

Additional  performanee  evaluation  ean  be  found  in  [6] .  For  example,  we  eonsider  the  effeet  of 
stale  group  size  information  on  the  performanee  of  the  balaneed  mode.  We  also  eonsider  the  ability 
of  the  probabilistie  elosest  algorithm  to  reaeh  the  elosest  reeeivers. 

3.4  Application  Scenarios 

Traditional  anycast  service  has  received  much  attention  because  of  its  applicability  for  selection 
from  a  set  of  replicated  servers.  As  services  become  more  complex  and  involve  distributed  groups 
of  servers,  that  the  ability  to  select  a  customized  subset  of  servers  (as  supported  by  PAMcast)  will  be 
of  increasing  interest.  Within  our  own  work,  we  have  explored  the  use  of  PAMcast  in  a  multi-media 
stream  caching  environment,  to  support  the  distribution  of  variable  numbers  of  copies  of  objects 
into  independent  caches  [5].  The  number  of  copies  depends  on  an  estimate  of  the  popularity  of 
the  object.  The  balanced  mode  is  most  appropriate,  since  a  global  goal  is  to  distribute  the  contents 
evenly  over  the  set  of  caches. 

4  Concast 

The  IP  multicast  communication  paradigm  lets  a  single  address  represent  a  set  of  nodes.  Nodes 
signal  the  network  their  requests  to  be  included  in  the  set  associated  with  a  particular  address;  the 
network  takes  responsibility  for  delivering  a  packet  addressed  to  that  address  to  all  nodes  in  the  set. 
This  multicast  “abstraction”  is  truly  scalable  in  that  it  minimizes  network  traffic  and  simplifies  fhe 
programming  model:  a  sender  can  freaf  any  number  of  receivers  as  a  single  enfify,  wifhouf  knowing 
abouf  individuals  or  even  how  many  receivers  fhere  are. 

Mainfaining  fhis  absfraclion  mechanism  when  informalion  has  fo  flow  from  fhe  group  back 
foward  a  single  node  requires  an  analogous  many-to-one  channel:  one  fhaf  enables  a  receiver  fo 
communicafe  wifh  an  arbifrary  number  of  senders  as  if  fhey  were  a  single  enfify.  In  fhe  absence  of 


such  a  channel,  group  applications  historically  have  resorted  to  ad-hoc  unicast-based  solutions  that 
limit  scalability.  The  purpose  of  the  concast  service  is  to  provide  such  a  many-to-one  channel. 

Concast  embodies  the  principles  of  anonymity  and  specialization  exemplified  by  multicast.  In 
particular,  it  preserves  anonymity  when  multicast  receivers  need  to  send  feedback  to  the  multicast 
sender.  Concast  allows  the  application  programmer  to  supply  a  merge  specification,  which  describes 
how  to  combine  or  aggregate  messages  inside  the  network,  as  they  travel  from  the  many  (senders) 
to  the  one  (receiver).  Programmability  is  crucial  to  the  utility  of  concast,  because  the  nature  of  the 
merging  operation  depends  upon  the  semantics  of  the  application’s  messages — unlike  the  simple 
generic  operation  provided  by  multicast  (i.e.  duplication),  which  is  useful  to  many  applications.  At 
the  same  time,  the  merge  specification  fits  into  a  specialized  interface  designed  specifically  fo  limit 
the  resources  available  for  processing  each  packet.  For  example,  the  merge  specification  template 
ensures  that  at  most  one  packet  is  forwarded  for  each  incoming  packet.  The  remainder  of  this  section 
provides  a  brief  overview  of  our  service  and  its  implementation.  For  a  more  detailed  description  of 
the  concast  service,  the  interested  reader  is  referred  to  the  individual  publications  [2,  1,4]. 

4.1  The  Concast  Service  and  Programming  Interface 

Concast  messages  are  sent  from  a  group  of  senders  to  a  single  receiver,  R.  A  concast  flow  is 
associated  with  a  pair  (G,  R),  where  G  is  the  group  identifier.  The  packefs  delivered  fo  ii  are  a 
funcfion  of  fhe  packefs  senf  by  fhe  members  of  G.  Concasf  packefs  are  ordinary  IP  dafagrams  wifh  R 
in  fhe  desfinafion  field  and  G  in  fhe  IP  options  field  (in  a  Concast  ID  opfion).  The  IP  source  address 
carries  fhe  unicasf  address  of  fhe  lasf  concasf  capable  roufer  fhaf  processed  fhe  packef.  Concasf 
capable  roufers  infercepf  and  diverf  for  processing  all  packefs  fhaf  use  fhe  Concasf  ID  opfion. 


getTagim):  a  tag  extraction  function  returning  a  hash  or  key  identifying  the  message.  Messages 
m  and  m'  are  eligible  for  merging  iff  getTag{m)  =  getTag{m'). 
merge(s,  m,  /):  the  function  that  combines  messages  together.  The  first  parameter  is  the  current 
merge  state  (i.e.,  information  representing  messages  that  have  already  been  processed).  The 
second  parameter  is  the  incoming  message  to  be  merged  into  the  state  s.  The  third  parameter 
is  the  “flow  state  block”  containing  information  about  the  concast  flow  to  which  m  belongs. 
done(s):  the  predicate  that  checks  s,  the  current  merge  state,  and  decides  whether  a  message 
should  be  constructed  (by  calling  buildMsg)  and  forwarded  to  the  receiver. 
buiMMsgis):  the  message  construction  function,  which  takes  the  current  merge  state,  s,  and  re¬ 
turns  the  payload  to  be  forwarded  toward  the  receiver,  along  with  the  updated  state. 


Figure  4:  Merge  Specification  Methods. 


The  concast  abstraction  allows  applications  to  specify  the  mapping  from  sent  messages  to  de¬ 
livered  message(s),  which  is  carried  out  in  the  concast-capable  routers  along  the  paths  from  senders 
to  R.  This  mapping  is  called  the  merge  specification-,  it  controls  (1)  the  relationship  between  the 
payloads  of  sent  and  received  datagrams,  (2)  the  timing  of  message  delivery,  and  (3)  packet  identifi¬ 
cation  (i.e.,  which  packets  are  merged  together).  The  merge  specification  is  defined  in  ferms  of  four 
custom  mefhods  or  funcfions  (see  Figure  4),  which  are  invoked  from  a  generic  packef-processing 
loop  (see  Figure  5),  which  is  fhe  same  for  each  flow.  This  generic  packef-processing  loop  is  invoked 
for  each  incoming  packef  belonging  fo  fhe  flow. 

The  concasf  framework  allows  users  fo  supply  fhe  definifions  of  fhe  cusfom  mefhods  using  a 
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ProcessPkt (Receiver  R,  Group  G,  IPDatagram  m)  { 
FlowStateBlock  fsb; 

DECTag  t; 

MergeStateBlock  s; 

fsb  =  LOOKUP_FLOW (R, G) ; 
if  (fsb  !=  NULL)  { 

t  =  f sb . getTag (m) ; 

S  =  GET_MERGE_STATE (fsb,t) ; 
s  =  UPDATE_TTL (s, m) ; 
s  =  f sb . merge ( s , m, fsb) ; 
if  ( f sb . done  ( s ) )  { 

(s,m)  =  fsb.buildMsg(s); 

FORWARD_DG (f sb, s, m) ; 

} 

PUT_MERGE_STATE (fsb, S , t ) ; 

} 

} 


Figure  5 :  Network  per-paeket  proeessing. 


(restrieted)  mobile-eode-language.  The  definitions  are  provided  by  the  applieation  reeeiver  and 
pulled  down  through  the  network  to  the  appropriate  nodes  in  a  “just-in-time”  manner,  as  senders 
join  the  group.  The  Concast  Signaling  Protocol  is  responsible  for  this  deployment.  For  eaeh  eoneast 
flow  (G,  R),  eaeh  eoneast-enabled  router  on  the  regular  unieast  path  from  some  member  of  G  to  i? 
maintains  the  following  information  in  a  flow  state  block  (FSB): 

Upstream  Neighbor  List  (UNL):  Eaeh  item  in  the  UNL  represents  a  eoneast-eapable  router  or 
sender  that  forwards  paekets  toward  R  along  a  path  that  goes  through  this  node.  Eaeh  entry  in 
the  UNE  list  eontains  a  node  identifier  and  a  soffsfate  timer.  The  softstate  timer  eontrols  the 
reelamation  of  the  resourees  assoeiated  with  the  entry;  if  it  expires  without  being  refreshed, 
the  entry  is  removed.  Thus  eaeh  eoneast-eapable  node  periodieally  sends  a  (unieast)  message 
downstream — that  is,  toward  the  flow’s  reeeiver — to  refresh  its  soft  state.  When  a  flow’s  UNE 
eontains  fewer  than  two  entries,  paekets  belonging  to  the  flow  are  forwarded  without  being 
proeessed. 

Merge  Specification:  the  definitions  of  the  getTagO,  merge(),  done(),  and  buildMsgO  funetions. 
Per-message  State  List:  A  list  of  in-progress  “merge  states”  indexed  by  message  tags,  i.e.  the 
intermediate  results  of  proeessing  ineoming  messages  belonging  to  this  flow. 

Ineoming  eoneast  paekets  are  elassified  into  flows  based  on  (G,  R).  If  no  ESB  is  found  for  a 
paeket,  it  is  disearded.  Otherwise  the  getTag  funetion  is  obtained  from  the  ESB  and  applied  to  the 
paeket  to  obtain  a  tag  that  identifies  fhe  equivalenee  elass  of  paekets  to  whieh  the  paeket  belongs. 
The  merge  funetion  is  next  invoked  on  the  eurrent  merge  state  for  the  tag  (i.e.,  the  merged  state  from 
all  messages  already  reeeived)  to  eompute  the  new  merge  state.  The  done  predieate  then  determines 
indieates  whether  the  merge  operation  is  eomplete.  If  so,  buildMsg  is  invoked  to  eonstruet  an 
outgoing  message  from  the  merged  state  that  is  forwarded  toward  the  reeeiver. 
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4.2  Concast  Signaling 

The  Concast  Signaling  Protocol  (CSP)  establishes  eoneast-related  state  in  network  nodes.  The 
protoeol  works  by  “pulling”  the  merge  spee  from  the  reeeiver  towards  senders  as  they  join  the  group; 
eoneast  flow  state  is  thus  initialized  along  the  path  from  eaeh  sender  to  the  reeeiver.  The  reeeiving 
applieation  initially  installs  the  merge  speeifieation  loeally  (i.e.  at  the  reeeiving  host).  Senders  join 
the  group  by  transmitting  a  Request  for  Merge  Spec  (RMS)  message  toward  the  reeeiver  R.  All 
CSP  messages  are  sent  as  regular  IP  unieast  datagrams  with  CSP  identified  in  the  protoeol  field, 
(G,  R)  in  the  Coneast  ID  option  field,  and  the  IP  Router  Alert  Option  (RFC  21 13)  to  flag  the  paeket 
for  hop-by-hop  proeessing  at  eoneast-eapable  routers.  Coneast-eapable  nodes  proeess  every  CSP 
paeket  as  deseribed  below. 

Whenever  a  CSP  message  is  sent  downstream,  the  IP  destination  address  is  R  (the  flow’s  re¬ 
eeiver);  CSP  messages  sent  upstream  have  the  IP  address  of  the  next  upstream  router  in  the  tree  as 
destination.  The  souree  address  of  a  CSP  paeket  is  always  the  (unieast)  IP  address  of  the  paeket 
originator,  whieh  may  be  a  router.  ^ 

Eaeh  CSP  message  eontains  a.  flow  id  {R,  G)  in  an  option  in  the  IP  header;  R  is  the  already- 
mentioned  reeeiver  address  and  G  is  the  eoneast  group  address.  In  addition,  eaeh  message  eontains 
an  indieation  of  the  message  type  and  type-speeifie  information.  The  message  types  along  with  their 
type-speeifie  information  are: 

Join  Flow  Request  (JFR):  This  message  is  transmitted  by  a  sender  toward  the  reeeiver.  It  informs 
the  nearest  downstream  eoneast-eapable  router  of  the  existenee  of  a  sender  for  this  flow. 

Concast  Join  Succeeded  (CJS):  This  message  is  sent  in  response  to  a  JFR,  after  the  downstream 
node  has  reeeived  the  merge  spee  and  set  up  the  flow. 

Request  for  Merge  Specification  (RMS):  This  message  is  always  sent  downstream.  It  indieates 
to  a  downstream  node  the  existenee  of  a  eoneast-eapable  router  upstream,  and  requests  that 
the  upstream  node  be  added  to  the  speeified  flow.  RMS  messages  eontain  a  Refresh  Flag, 
whieh  indieates  whether  the  message  is  being  sent  to  keep  an  entry  alive  in  the  downstream 
UNL.  If  the  flag  is  eleared,  the  sender  expeets  the  flow’s  merge  speeifieation  to  be  sent  in 
response  to  the  message. 

Merge  Specification  (MS):  The  MS  message  is  sent  upstream,  in  response  to  an  RMS  message 
(with  the  Refresh  Flag  elear),  and  earries  the  merge  speeifieation  for  the  requested  reeeiver 
and  group. 

Error:  The  Error  message  eonveys  information  about  various  error  eonditions  related  to  the  eon¬ 
east  data  flow  (e.g.  destination  unreaehable).  It  may  be  sent  to  the  originator  of  a  RMS 
message  or  to  all  the  members  on  the  Upstream  Neighbor  Fist. 

The  normal  progression  of  messages  in  the  CSP  protoeol  is  illustrated  in  Figure  6.  Initially 
reeeiver  R  indieates  to  the  loeal  eoneast  implementation — labeled  “RCSPD”,  for  receive-CSP  dae¬ 
mon  in  the  figure — its  request  to  ereate  (and  reeeive  from)  the  flow  [R,  G),  providing  a  merge 
speeifieation  as  part  of  the  request.  The  RCSPD  ereates  the  FSB  for  {R,  G)  with  an  empty  UNF. 

'This  is  a  crucial  from  our  early  designs,  which  specified  that  the  concast  group  ID  he  used  as  the  source  ID.  The 
change  was  necessary  to  enable  connectivity  problems  in  the  concast  tree  to  be  detected  via  ICMP.  (ICMP  messages 
cannot  be  sent  in  response  to  packets  with  concast  source  addresses.) 
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Figure  6:  Concast  Signaling  Message  Flow. 


Subsequently,  sender  (Sq)  indicates  to  its  local  SCSPD  (send-CSP  daemon)  its  request  to  join 
the  group  {R,  G).  The  SCSPD  then  transmits  a  Join  Flow  Request  message  with  source  address  Sq 
and  destination  R.  At  the  first  concast-capable  node  on  the  path  between  Sq  and  R  (labeled  Nq  in 
the  figure),  the  JFR  message  is  processed  by  the  RCSPD,  which  creates  a  FSB  for  the  flow,  with  a 
UNL  containing  only  ^o;  the  state  of  the  flow  is  marked  “pending”,  to  indicate  that  it  is  not  yet  fully 
established.  Node  Nq  then  sends  a  RMS  message  toward  R.  As  the  figure  indicates,  non-concast- 
capable  nodes  (JVi)  simply  forward  the  CSP  messages  normally.  At  the  next  concast-capable  hop 
{N2),  again  a  FSB  is  created  and  another  RMS  message  is  sent.  When  it  reaches  R,  N2  is  added  to 
the  UNL  for  [R,  G)  and  the  complete  merge  specification  is  sent  back  upstream  to  N2,  the  source  of 
the  RMS.  When  N2  receives  the  MS  message,  it  installs  the  merge  specification,  sets  the  flow  state 
to  FORWARDING,  and  then  sends  the  MS  message  directly  to  Nq  (because  it  is  in  the  UNL).  Finally, 
after  installing  the  merge  specification,  the  RCSPD  at  Nq  returns  a  CJS  message  to  Sq,  at  which 
point  the  original  “join  group”  request  succeeds,  and  the  sender  is  free  to  transmit  data  on  the  flow. 
Processing  of  concast  datagrams  belonging  to  the  (i?,  G)  flow  is  now  being  performed  at  Nq  and 
N2,  but  only  one  sender  belongs  to  the  flow,  so  any  messages  sent  by  Sq  are  forwarded  unchanged 
to  R. 

When  another  sender  joins  the  same  flow,  the  CJS/RMS  messages  progress  downstream  only 
until  they  reach  a  node  that  is  already  a  member  of  the  flow;  that  member  transmits  the  merge 
specification  back  upstream,  extending  the  flow  tree  to  the  new  sender. 

4.3  Application  Scenarios  and  Benefits 

Merging  packet  flows  inside  the  network  offers  several  benefits.  Some  have  already  been  described 
when  we  discussed  the  principle  of  anonymity,  above.  Here  we  present  some  other  example  benefits. 

Implosion  Avoidance:  Packet  implosion  occurs  when  a  large  number  of  packets  must  be  received 
and  processed  by  the  destination  over  a  short  interval.  As  the  number  of  senders  increase, 
buffer  space  requirements  at  the  receiver  and  the  processing  load  on  the  receiver  increases. 
This  may  ultimately  result  in  lost  packets  at  the  receiver  and  an  overall  drop  in  throughput. 
Implosion  is  especially  a  problem  in  reliable  multicast,  which  is  useful  for  information  dis¬ 
semination  applications. 

Reduced  Bandwidth:  Merging  messages  together  near  their  point  of  origin  (i.e.,  near  the  senders) 
can  substantially  reduce  the  traffic  load  imposed  on  the  network.  This  is  particularly  impor- 
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tant  in  WAN  settings  such  as  the  Internet  where  bandwidth  is  shared,  and  thus  is  a  valuable 
resource.  An  example  of  an  application  to  which  this  benefit  is  applicable  is  an  audio-video 
conference,  in  which  some  participants  are  behind  limited-bandwidth  links.  By  combining 
audio  and  video  streams  in  the  network  (see  the  next  section),  the  conference  remains  fully 
distributed  and  each  participant  sees  a  single  composite  stream. 

Larger  Packets:  Router  performance  is  typically  given  in  terms  of  packets/second,  rather  than 
bits  per  second.  In  other  words,  small  packets  present  more  of  a  performance  challenge 
than  large  ones.  By  concatenating  or  merging  multiple  small  messages  into  a  larger  single 
message,  the  fixed  per-packef  cosf  can  be  amortized  over  a  larger  (aggregate)  packef.  For 
applicafions  in  which  large  numbers  of  small  packefs  fravel  from  many  senders  toward  a 
server — for  example,  TCP  acknowledgemenf  packefs  fraveling  foward  a  busy  web  server — 
our  simulation  sfudy  showed  fhaf  under  cerfain  condifions,  aggregafing  packefs  can  improve 
fhroughpuf  [3]. 

A  variefy  of  group-orienfed  applicafions  can  benefif  from  concasf.  One  class  of  such  applica¬ 
tions  are  group-request-reply  applicafions,  in  which  a  single  machine  issues  a  requesf  fo  a  group  of 
machines  (fypically  via  mulficasf)  and  fhen  wails  for  a  response  from  all  group  members  (histori¬ 
cally  implemenled  via  unicasl).  This  model  of  inferaclion  is  used  in  nelwork  Iransporl  prolocols, 
and  in  applicafion-level  communicalion.  For  example,  sender-initialed  reliable  mulficasf  prolocols 
Iransmil  dala  lo  receivers  via  mulficasf  and  fhen  waif  for  receivers  lo  send  ACK  messages  lo  Ihe 
sender.  Similarly,  application-level  concasf  ACK  messages  are  required  by  a  variefy  of  applica¬ 
tions  lo  ensure  end-lo-end  reliabilily.  Olher  applicafions  need  lo  galher  dislribuled  slafe  informalion 
before  making  a  decision.  Dislribuled  consensus  algorilhms  may  requesf  a  vote  from  all  members. 

Two  olher  classes  lhal  exhibil  concasf  communication  patterns  are  client-multiserver  and  multiclient- 
server  applications.  To  ensure  reliability  or  availability,  client-multiserver  applications  replicate 
server  functionality  across  multiple  machines.  Client  requests  are  multicasted  to  the  servers  who 
send  simultaneous  responses  to  the  client.  The  client  often  needs  only  one,  or  K  out  of  N,  re¬ 
sponses.  For  example,  a  distributed  database  may  require  multiple,  K  out  of  N  ACKs  when  storing 
replicas  of  a  newly  written  record.  Multiclient-server  refers  to  the  classical  client-server  model, 
in  which  a  single  server  responds  to  requests  from  any  number  of  client  machines.  In  this  model, 

N  clients  transmit  request  messages,  often  simultaneously,  to  a  single  server.  In  many  cases,  the 
requests  are  similar  or  even  the  same.  Consider  a  web  server  at  a  popular  web  site.  It  can  service 
hundreds  or  thousands  of  requests  each  second,  often  for  the  same  page.  Logically  these  requests 
can  be  viewed  as  concast  communication. 

Another  class  of  applications  that  are  characterized  by  concast  communication  patterns  is  the 
report-in  style  applications  in  which  distributed  machines,  sensors,  robots,  or  other  devices  peri¬ 
odically  transmit  data  to  a  centralized  control  or  monitoring  system.  In  some  cases,  transmission 
from  the  concast  senders  may  be  continuous  such  as  stereo  video  feeds  sent  to  a  centralized  video 
processing  engine  that  may  reconstruct  3D  models,  extracts  depth,  perform  motion  detection,  target 
tracking,  etc. 

Distributed  sensor  networks  represent  another  class  of  applications  that  can  benefit  from  con¬ 
cast.  Many  of  the  current  systems  rely  on  a  scheme  in  which  data  produced  by  the  distributed 
sensors  is  collected  at  a  single  node  that  performs  image  processing,  interpretation,  and  display  of 
the  data.  As  the  number  of  sensors  in  the  system  grows,  some  form  of  hierarchy  is  needed  to  keep 
the  central  server  (and  its  incoming  channels)  from  becoming  overloaded.  Using  a  concast  service 
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to  aggregate  or  filter  data  en  route  to  the  eolleetion  node  automatieally  delegates  proeessing  into  the 
network. 

4.4  Securing  the  Concast  Service 

Applications  benefit  from  concast  by  being  able  to  apply  merge  processing  at  the  location  in  the 
network  where  it  provides  maximum  leverage — where  packet  streams  converge.  However,  this 
requires  that  end  users  trust  the  infrastructure  to  perform  certain  critical  operations,  namely  (i) 
admit  users  to  the  concast  session,  and  (ii)  examine  and  possibly  modify  user  data.  In  some  cases 
users — in  particular  the  receiver,  which  presumably  will  be  making  use  of  the  received  data — may 
not  trust  some  portions  of  the  infrastructure  to  perform  these  functions  securely.  Unfortunately,  the 
use  of  end-to-end  security,  which  is  the  standard  practice  when  untrusted  nodes  handle  user  data, 
is  not  compatible  with  merging  data  inside  the  network.  Conversely,  it  is  likely  that  only  a  subset 
of  users — say,  those  who  have  paid  for  the  privilege — will  be  authorized  to  participate  in  concast 
sessions.  Thus,  we  need  to  extend  the  concast  service  described  above  with  mechanisms  to  enforce 
various  policies  that  govern  which  nodes  can  participate  in  a  concast  session. 

With  this  in  mind,  we  developed  a  secure  concast  design  based  on  the  principle  that  control 
over  participation  in  a  concast  flow  (and  thus  access  to  user  data)  is  equivalent  to  possession  of  the 
merge  specification.  In  other  words,  any  node  possessing  the  merge  specification  will  be  able  to  get 
hold  of  and  possibly  modify  user  data  and  forward  it  to  the  receiver,  and  nodes  without  the  merge 
specification  will  not  be  able  to.  It  follows  that  the  security  of  a  flow  resfs  upon  fhe  secrecy  of  fhe 
merge  specificalion.  This  has  fwo  imporfanf  consequences: 

•  The  propagation  of  fhe  merge  specification  is  confrolled  by  fhe  signaling  profocol;  fhus  if 
musf  be  modified  fo  ensure  fhaf  only  frusfed  nodes  (i.e.  nodes  aufhorized  according  fo  fhe 
policies  menfioned  above)  are  able  fo  obfain  fhe  merge  specification. 

•  Given  secure  disfribufion  of  fhe  merge  spec,  securify  of  user  dafa  can  be  delegafed  fo  fhe  merg¬ 
ing  process.  Thai  is,  mechanisms  (including  secref  keys)  needed  fo  ensure  confidenlialily  and 
aulhenficify  of  user  data  senf/received  on  fhe  flow  can  be  placed  in  fhe  merge  specification. 

This  resulfs  in  a  nice  separation  of  concerns:  fhe  applicafion  is  responsible  for  fhe  security 
of  ifs  own  dafa  en  route,  via  fhe  merge  spec  if  supplies;  fhe  concasf  infraslruclure  is  responsible 
for  ensuring  fhaf  only  duty  aufhorized  nodes  parficipafe  in  processing  fhaf  user  dafa  according  fo 
fhe  given  specificafion.  In  parficular,  applicafions  can  choose  fo  employ  “null”  mechanisms  in  fhe 
merge  spec  if  fhey  are  nol  concerned  wifh  security. 

Implemenfing  secure  concasf  required  subsfanfial  modificafions  fo  fhe  signaling  protocol,  as 
well  as  more  modesl  modifications  fo  fhe  generic  merging  framework.  We  describe  fhe  changes  to 
fhe  signaling  profocol  firsf. 

4.4.1  Modifications  to  the  Signaling  Protocol 

The  secure  concast  signaling  protocol  is  required  to  enforce  the  following  policies: 

•  User  specification  of  which  senders  (users)  are  authorized  to  join  a  given  flow.  This  policy  is 
supplied  by  the  receiver  as  part  of  the  merge  specification. 
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•  User  specification  of  which  routers  are  authorized  to  participate  in  the  flow,  i.e.  to  carry  out 
the  merge  specification  on  user  data  sent  toward  the  receiver.  This  policy  also  is  supplied  by 
the  receiver  in  the  merge  spec. 

•  Service  provider  specification  of  which  users  are  authorized  to  create  or  join  concast  flows. 
Since  concast  is  a  “value-add”  service,  providers  want  to  ensure  that  only  customers  who  have 
paid  for  the  service  have  access  to  its  benefits.  This  policy  is  assumed  to  be  installed  locally 
at  each  router  in  the  network. 

•  Service  provider  specification  of  which  routers  are  permitted  to  participate  in  concast  flows. 
Providers  may  or  may  not  trust  each  others’  routers.  For  example,  a  transit  provider  may  only 
accept  merge  specifications  from  (or  supply  merge  specifications  to)  routers  belonging  to 
other  providers  with  which  it  has  “concast  peering  agreements”.  This  policy  is  also  assumed 
to  be  installed  locally  at  each  router. 

Enforcing  these  policies  is  a  challenge  because  the  point  of  concast  is  to  preserve  anonymity. 
Requiring  the  receiver  to  make  an  explicit  authorization  decision  on  each  sender  who  wants  to  join 
a  flow  would  eliminate  the  scalability  benefits  that  motivate  the  use  of  concast  in  the  first  place.  The 
only  way  around  this  dilemma  is  to  rely  on  transitive  trust:  the  receiver  has  to  rely  on  upstream 
routers  to  enforce  its  policies.  Similarly,  intermediate  nodes  rely  on  routers  upstream  to  enforce 
their  policies  about  who  can  be  a  sender  or  merge  point.  It  follows  that  to  join  the  flow,  nodes  must 
not  only  be  trusted  to  handle  user  data,  they  must  also  be  trusted  to  enforce  all  relevant  policies. 

Since  the  security  of  the  flow  rests  on  the  secrecy  of  the  merge  specification,  and  being  trusted 
to  join  the  flow  implies  being  trusted  to  enforce  all  relevant  policies,  policies  can  simply  be  carried 
in  the  merge  specification.  Thus,  the  main  modifications  required  in  the  signaling  protocol  are: 

1 .  Signaling  messages  are  modified  to  carry  origin  authentication  information.  In  particular,  JFR 
messages  carry  a  authenticator  for  the  joining  sender;  RMS  messages  carry  an  authenticator 
for  the  requesting  router  in  addition  to  the  sender  credential  from  the  original  JFR  message. 
MS  messages  carry  authentication  information  for  the  receiver  (i.e.  origin  of  the  merge  spec¬ 
ification)  and  the  downstream  neighbor  router  (if  any)  originating  the  message.  We  do  not 
specify  a  particular  authentication  mechanism;  also,  participants  are  assumed  to  obtain  any 
secrets  needed  to  verify  authenticity  through  out-of-band  means.  (Our  implementation  sup¬ 
ports  both  shared-secret  and  public-key  authentication  mechanisms.) 

2.  Before  establishing  flow  state  on  behalf  of  an  upstream  node  (in  response  to  a  JFR  or  RMS 
message),  a  router  applies  all  relevant  policies.  In  particular,  when  a  JFR  or  RMS  message 
is  received,  the  local  node  policy  is  applied  to  the  (authenticated  identity  of  the)  requesting 
upstream  node  before  establishing  the  FSB  and  forwarding  the  request  downstream.  When/if 
a  MS  message  is  received  in  reply,  the  policies  in  the  merge  spec  are  then  applied  to  the  node 
in  the  UNF,  and  only  if  it  is  authorized  is  the  flow  finally  established. 

3.  To  keep  the  merge  specification  secret  in  transit  between  authorized  concast-capable  nodes, 
a  step  is  added  to  the  protocol  which  establishes  an  IPsec  tunnel  between  a  node  and  its 
upstream  neighbor  before  the  MS  message  is  transmitted.  All  concast  signaling  messages 
between  the  two  routers  are  transmitted  over  this  tunnel,  which  is  shared  by  all  flows  for 
which  the  two  routers  are  neighbors. 
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These  modifications  result  in  a  somewhat  more  complex  signaling  process.  This  is,  however,  re¬ 
quired  to  maintain  the  invariant  that  every  participating  node  is  acceptable  according  to  its  neigh¬ 
bors’  (in  the  concast  flow  tree)  policies. 

4.4.2  Merge  Framework  Modifications 

In  addition  to  changes  required  to  secure  the  “control  plane”,  changes  were  also  needed  in  the 
merging  framework  in  order  to  support  user-defined  encryption/decryption  in  the  data  plane. 

First,  we  modified  the  merge  specification  to  include  user-defined  encryption  function  and  de¬ 
cryption  functions,  as  well  as  the  secret  key  to  be  used  for  encryption  and  decryption.  These  func¬ 
tions  may  be  specified  via  Java  byte  codes,  or  by  reference  to  standard,  predefined  encryption  and 
decryption  functions  (e.g.  AES-ECB),  which  we  made  part  of  the  standard  merging  framework  (i.e. 
merged)  implementation.  We  also  modified  the  generic  message-processing  loop  shown  in  Eig- 
ure  5  to  invoke  these  functions  to  decrypt  each  incoming  concast  data  message,  and  to  encrypt  each 
outgoing  concast  data  message.  (Applications  that  do  not  wish  to  secure  their  data  can  specify  stan¬ 
dard  “null”  encryption  methods.)  As  part  of  the  encryption  specification,  the  framework  allows  the 
user  to  specify  whether  a  MAC  (message  authentication  code)  should  be  included  in  the  encrypted 
message.  If  so,  the  MAC  will  be  checked  when  the  packet  is  decrypted  to  verify  its  integrity. 

The  second  change  to  the  framework  allows  the  per-tlow  merge  daemon  at  a  merging  node  to 
distinguish  among  senders,  merging  nodes,  and  the  receiver.  Merge  daemons  executing  on  sender 
nodes  receive  packets  over  a  local  socket.  Because  they  arrive  unencrypted,  the  decryption  function 
does  not  need  to  be  invoked  (only  the  encryption  must  be  called).  On  receiver  nodes,  incoming 
packets  are  processed  and  delivered  straight  to  the  receiver  application.  In  this  case,  only  the  decrypt 
function  is  required.  On  intermediate  nodes  both  encryption  and  decryption  are  invoked  on  all 
incoming  and  outgoing  packets  (assuming  the  flow  is  in  the  merging  state). 

Because  the  sending  and  merging  operations  are  controlled  by  separate  policies,  the  first  down¬ 
stream  node  from  the  sender  responds  to  the  JER  message  with  a  partial  merge  specification,  con¬ 
taining  only  the  secret  key  and  the  encryption  function;  in  particular,  it  does  not  contain  the  merge 
functions. 

4.5  Implementation  and  Results 

We  have  implemented  concast  in  the  Einux  operating  system.  Our  implementation  has  three  com¬ 
ponents: 

CSP  Daemon 

The  CSP  daemon  (CSPd)  is  responsible  for  running  the  concast  signaling  protocol  (CSP),  and 
maintaining  the  concast  state  on  routers.  Two  types  of  the  CSPd  are  implemented,  one  for  the 
receiver,  one  for  the  sender.  The  sender  CSPd  runs  on  sender  nodes  to  extend  and  maintain  the 
concast  session  and  pull  the  merge  specification  down  along  the  concast  path.  The  receiver 
CSPd  runs  on  the  receiver  and  concast-capable  nodes  along  the  concast  path.  It  is  responsible 
for  spawning  the  merge  daemon  (MERGEd)  and  signaling  for  the  merge  specification  to  be 
downloaded.  The  implementation  of  the  two  CSPd’s  consists  of  approximately  10,000  lines 
of  C-i-i-  code. 

Merge  Eramework 
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The  merge  daemon  (MERGEd)  that  is  spawned  by  CSPd  on  the  concast  path  is  responsible 
for  locating  and  applying  user-defined  merge  specification  to  concast  packets.  Currently,  the 
merge  specification  defined  in  Eig.  4  can  be  implemented  in  Java  or  Tel.  The  MERGEd  is 
implemented  in  slightly  more  than  2,000  lines  of  Java  code. 

Kernel  Modification 

A  concast  kernel  module  supports  socket  options  that  allow  users  to  mark  sockets  as  concast 
sockets,  and  to  join  and  leave  concast  groups.  It  also  supports  the  intra-node  communication 
between  merge  daemons  and  CSP  daemon.  The  packet  path  is  modified  so  that  concast  pack¬ 
ets  are  recognized  and  shunted  to  the  appropriate  daemon  for  processing.  The  concast  module 
implementation  consists  of  less  than  3,000  lines  of  code. 

We  have  developed  two  test  applications  on  this  concast  implementation:  video  merging  and 
audio  merging.  In  the  video  merging  application,  we  emulate  a  distant  learning  environment,  where 
the  instructor  sees  the  video  feed  from  each  student,  while  each  student  only  needs  to  see  the  in¬ 
structor.  In  this  type  of  application,  the  video  flows  from  sfudents  fo  insfrucfor  (who  is  behind  a 
modesf-capacity  link)  result  in  implosion  and  poor  video  quality  if  the  bandwidth  is  not  managed 
carefully.  Using  concast,  flows  can  be  merged  and  thinned  in  a  manner  similar  to  that  used  with 
real-time  protocol  mixers,  but  in  just  the  locations  in  the  network  where  the  processing  is  needed 
(as  opposed  to  specialized  servers,  which  may  be  off  the  normal  path  between  participating  nodes). 
The  audio  merge  function  is  similar:  audio  streams  from  different  sources  are  combined  and  played 
out  at  a  single  location  that  is  behind  a  bottleneck  network  link;  this  could  be  used  for  a  field  briefing 
from  distributed  intelligence  sources,  or  for  performance  of  a  distributed  quartet. 

To  support  this  type  of  application,  we  designed  a  simple  merge  function  that  “thins”  incoming 
video  streams  by  downsampling  (i.e.  throwing  away  pixel  values  in  a  controlled  way)  .  It  combines 
packets  so  frames  from  different  incoming  streams  are  combined  into  a  single  composite  (tiled) 
frame  made  up  of  reduced-size  frames;  thus  the  outgoing  link  from  each  concast-capable  router 
carries  a  single  video  stream.  At  each  hop,  the  merge  function  keeps  track  of  the  number  of  incom¬ 
ing  streams  and  the  number  of  original  streams  contained  in  each.  It  then  assigns  a  region  of  the 
outgoing  frame  to  each  incoming  video  stream  and  down-samples  the  stream  appropriately  to  fit  in 
the  assigned  region.  As  new  students  join  the  class,  the  other  images  are  adjusted  to  make  room  for 
the  new  student  (see  Eigure  7).  Each  composite  stream  carries  information  about  how  many  original 
streams  it  contains,  and  how  they  have  been  last  combined  so  that  each  node  can  determine  how  to 
combine  its  incoming  streams. 

We  measured  the  load  incurred  on  the  routers  while  performing  the  video  merge  on  a  single 
concast  flow.  The  experimental  topology  is  shown  in  Eigure  8.  We  used  four  video  senders,  each 
transmitting  a  baseband  video  stream  of  five  320x240  frames  per  second,  using  8-bit  grayscale,  for 
a  rate  of  about  3  Mbps. 

Eigure  9  shows  the  CPU  load  caused  by  merge  processing  on  each  of  the  three  merging  routers 
(nodes  1,  2,  and  3)^  Our  concast  merge  specification  was  written  in  Java  and  runs  in  a  user-level 
JVM,  which  accounts  for  the  majority  of  the  load.  Initially,  video  sources  are  started  on  senders  1 
and  3.  This  causes  the  the  load  on  merging  node  1  to  increase  (see  point  A  in  the  graph).  Video 

^Merging  node  1  happened  to  be  a  600  MHz  Pentium,  whereas  the  other  nodes  were  400  MHz  Pentiums  -  so  the  load 
appears  to  be  lower  on  node  1  even  though  it  is  doing  the  same  processing  as  the  other  nodes. 
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(a)  Initially  there  may  only  be  four  students  whose 
video  is  merged  into  a  single  video  stream  that  is 
displayed. 


(b)  As  more  people  join  the  class,  the  concast 
merge  function  dynamically  adjusts  the  video  to 
make  room  for  the  new  students. 


Figure  7:  Illustration  of  a  Distance  Learning  Application:  The  video  stream  from  each  source  is 
down-sampled  at  the  merge  point,  resulting  in  the  viewing  window  size  at  the  receiver  remaining 
constant  while  the  number  of  sub-windows  increases  (i.e.  more  participants),  and  the  size  of  sub¬ 
windows  decreases. 


Figure  8:  Topology  used  in  the  video  merging  experiments. 
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CPU  Usage  01  Concast  Merge  Nodes 


Figure  9:  Merge  processing  load  imposed  on  concast  routers. 


sources  were  then  started  on  senders  2  and  4  (points  B  and  C),  causing  the  load  to  increase  on  merg¬ 
ing  nodes  2  and  3  respectively^.  Note  that  when  a  node  only  has  one  upstream  neighbor,  packets  are 
forwarded  as  normal  without  invoking  the  merge  processing.  Despite  being  implemented  in  Java, 
the  merging  code  (which  is  merging  two  3  Mbps  incoming  streams  into  a  single  outgoing  3  Mbps 
stream)  does  not  exceed  a  60%  CPU  load.  At  the  end  of  the  test,  the  senders  terminate  (points  D,  F, 
and  I)  and  after  the  concast  softstate  times  out,  the  CPU  loads  again  return  to  zero  (points  E,  G,  and 
J). 

To  quantify  the  cost  of  security  for  concast  service,  we  ported  the  video-merging  application  to 
the  secure  concast  framework,  and  measured  the  load  at  an  intermediate  node  (1.4  GHz  Pentium) 
where  two  8-frames-per-second  streams  were  being  merged  into  a  single  stream.  The  graphs  in 
Figure  10  show  the  results.  The  user  component  of  the  overall  load  increased  from  around  45% 
using  the  “null”  ciphersuite  to  around  65%  when  the  Advanced  Encryption  System  was  used  with 
a  SHA-1 -based  MAC.  The  system  component  of  the  load  remained  more  or  less  constant  at  around 
10%. 


4.6  Discussion 

We  have  designed  and  implemented  a  secure  version  of  the  concast  protocol,  to  make  it  feasible 
to  deploy  the  service  in  production  networks  where  various  users  and  service  providers  may  not 
trust  each  other.  We  have  also  designed  and  implemented  a  means  of  specifying,  verifying  and 
merging  policies  using  address  prefixes.  The  mechanism  allows  the  policy  to  define  classes  of 
allowed/forbidden  nodes  based  on  IP  addresses. 

It  is  fairly  clear  that  the  need  to  specify  these  policies  represents  at  least  a  potential  violation  of 
the  principle  of  anonymity:  receivers  are  required  to  identify  specific  nodes  that  are  trusted  (or  not 
trusted).  Thus,  there  is  a  fundamental  conflict  between  access  control  and  anonymity,  which  implies 
that  for  services  to  be  anonymous,  they  must  be  “throwaway”,  that  is,  sufficiently  lightweight  to  be 
made  available  to  all  comers.  The  design  of  the  service  described  in  the  next  section  reflects  this 

^This  actually  reduces  the  load  on  merging  node  1  slightly  because  node  1  switches  from  horizontal  down-sampling 
to  vertical  down-sampling  (which  is  a  better  match  for  the  data  structures  used.)  When  senders  2  and  4  terminate  the  load 
returns  (point  H). 
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CPU  utilization  (8  FPS)  (NULL) 


CPU  Utilization  (8  FPS)  (AES/SHA1) 
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(a)  Load  vs.  time,  with  null  security 


(b)  Load  vs.  time,  with  AES/SHA-1 


Figure  10:  User  and  system  load:  video  merge  at  8  frames/see 


observation. 

5  Ephemeral  State  Processing 

Most  interesting  aetive  serviees  require  per-user  (or  per-applieation)  state  in  the  network.  Indeed, 
the  ability  for  paekets  to  exehange  and  eolleet  information  as  they  travel  through  the  network  is 
one  of  the  interesting  eapabilities  of  aetive  networking.  However,  the  need  to  set  up,  manage,  and 
reelaim  state  is  a  major  impediment  to  sealability.  For  example,  although  they  are  very  sealable  in 
terms  of  the  number  of  users  per  session,  neither  PAMeast  nor  Coneast  seales  espeeially  well  in  the 
“number  of  sessions”  dimension,  beeause  of  limits  on  the  amount  of  state  storage  available  at  eaeh 
node.  In  the  ease  of  Coneast,  we  addressed  this  problem  by  providing  meehanisms  to  eontrol  aeeess 
to  the  serviee. 

A  different  approaeh  led  to  a  novel  approaeh  to  aetive  networking  based  on  an  ultra-lightweight 
form  of  state,  whieh  has  essentially  zero  management  eost.  This  approaeh,  whieh  we  eall  ephemeral 
state  processing  (ESP),  is  simple  enough  to  be  implementable  direetly  in  hardware,  and  to  support 
packet  processing  at  line  speeds.  (In  other  words,  it  is  designed  to  scale  in  the  “number  of  simulta¬ 
neous  sessions”  dimension.)  ESP  allows  information  carried  in  packets  to  be  temporarily  stored  in 
the  network,  combined  with  information  from  other  packets,  and  forwarded  to  a  destination.  All  of 
this  occurs  under  direct  user  control,  with  no  out-of-band  signaling  or  control  setup  required. 

Processing  in  ESP  is  carried  out  in  short  computations  called  operations  that  are  initiated  by 
packets  as  they  pass  through  the  network.  Each  ESP-capable  node  supports  a  set  of  these  compu¬ 
tations,  which  may  update  the  node  state  and/or  fields  of  the  packet.  One  way  to  think  about  ESP 
computations  is  that  a  single  packet  initiates  a  spatial  sequence  of  operations — one  per  node  that 
forwards  it — while  a  series  of  packets  initiates  a  temporal  sequence  of  operations  at  a  single  node. 
Interesting  distributed  computations  can  be  constructed  by  judicious  arrangement  of  operation  se¬ 
quences  in  time  and  space. 

The  lightweight  nature  of  ESP  stems  from  the  simplicity  and  fixed  cosf  of  fhe  operations,  and 
from  fhe  facf  fhaf  stored  sfafe  persisfs  af  a  node  for  only  a  shorf,  fixed  fime — on  fhe  order  of  seconds. 


20 


Little’s  result  says  that  for  a  given  size  store,  the  maximum  rate  of  state  usage  that  ean  be  sustained 
is  inversely  proportional  to  the  holding  time.  Thus  by  halving  the  “holding  time”  for  state  storage, 
we  double  the  sustainable  rate  of  usage.  Beeause  the  resourees  eonsumed  by  eaeh  ESP  operation 
are  small  and  fixed,  there  is  no  need  for  aeeess  eontrol — we  believe  it  is  quite  feasible  for  nodes  to 
proeess  ESP  paekets  at  line  speeds. 

ESP  is  espeeially  useful  for  auxiliary  eomputations,  that  is,  proeessing  in  the  network  that  sup¬ 
ports  enhaneed  serviees.  In  partieular,  it  offers  a  solution  to  the  problem  of  determining  where 
enhaneed  funetionality  should  be  installed,  by  allowing  end  systems  to  extraet  a  limited  amount  of 
information  about  topology  from  the  network. 

In  the  next  seetions  we  deseribe  the  eomponents  of  ESP:  the  Ephemeral  State  Store,  the  opera¬ 
tions  initiated  by  paekets,  and  the  wire  protoeol. 

5.1  Ephemeral  State  Store 

Mueh  of  the  power  and  sealability  of  our  approaeh  stem  from  the  availability  of  an  assoeiative 
memory  ealled  an  ephemeral  state  store  at  eaeh  node.  An  assoeiative  memory  allows  fixed-size 
bif  sfrings  fo  be  assoeiafed  wifh  keys  or  tags  for  subsequenf  refrieval  and/or  updafe.  We  model  fhe 
sfore  as  a  sef  of  (tag,  value)  pairs;  eaeh  lag  has  al  mosl  one  value  bound  lo  if.  Bolh  fags  and  values 
are  fixed-size  bif  sfrings.  No  slruelure  is  imposed  on  eilher  lags  or  values  by  fhe  slale  sfore;  Iheir 
meaning  and  slruelure  is  defined  by  fhe  appliealions.  In  whal  follows,  x  denoles  an  arbilrary  lag, 
while  e  denoles  an  arbilrary  value. 

The  ephemeral  slale  store  is  aeeessed  Ihrough  Ihe  following  operalions: 

1.  put(x,  e):  bind  Ihe  value  e  to  Ihe  lag  x.  After  Ibis  operalion,  Ihe  pair  (x,  e)  is  in  Ihe  sel  of 
bindings  of  Ihe  store. 

2.  get(x):  Relrieve  Ihe  value  bound  to  lag  x,  if  any.  If  no  pair  (x,e)  is  in  Ihe  store  when 
Ibis  operalion  is  invoked,  Ihe  speeial  value  _L,  whieh  differs  from  every  legilimale  value,  is 
relumed. 

The  ephemeral  slale  store  has  Iwo  dislinelive  eharaelerislies.  The  firsl  is  lhal  bindings  are 
ephemeral:  eaeh  (lag,value)  pair  is  aeeessible  for  only  a  fixed  interval  of  lime  after  il  is  erealed. 
The  parameter  T^  is  Ihe  lifetime  of  a  binding  in  Ihe  store;  onee  erealed,  a  binding  remains  in  Ihe 
store  for  T;  seeonds  and  Ihen  vanishes.  Note  lhal  bindings  eannol  be  prevented  from  disappearing: 
Ihere  is  no  way  to  refresh  ephemeral  slate.  Also,  Ihe  value  of  T^  should  be  approximately  Ihe  same 
for  every  node,  so  lhal  users  ean,  when  neeessary,  be  assured  lhal  bindings  have  indeed  expired. 

The  imporlanee  of  Ihe  finite  lifelime  is  lhal  il  allows  Ihe  resouree  requirement  of  eompulalions 
using  Ihe  store  to  be  preeisely  bounded.  The  flip  side  of  Ihis  is  lhal  any  value  in  Ihe  store  musl  be 
relrieved  wilhin  Ihe  slate  lifelime  or  lost  Eor  sealabilily,  we  wanl  Ihe  value  of  T;  to  be  as  shorl 
as  possible;  for  robuslness,  il  needs  to  be  long  enough  for  inleresling  end-to-end  serviees  to  be 
eompleled.  A  delailed  deseriplion  of  how  Ihe  value  of  T^  is  sel  is  oulside  Ihe  seope  of  Ibis  paper. 
Eor  now  Ihe  reader  may  assume  lhal  il  is  on  Ihe  order  of  a  few  seeonds.  In  olher  words,  ephemeral 
slate  eompulalions  musl  eomplele  wilhin  a  few  round-lrip  limes  Ihrough  Ihe  nelwork. 

Il  is  imporlanl  to  note  lhal  Ihe  eapaeily  of  an  ephemeral  slate  store  is  determined  by  Ihe  amounl 
of  memory  available  al  Ihe  node.  We  wanl  Ihe  lag  spaee  to  be  large  enough  so  lhal  users  ean  choose 
tags  at  random  and  be  assured  lhal,  wilh  high  probabilily,  a  lag  ehosen  al  lime  t  will  nol  be  in  use  by 
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any  other  user  during  the  interval  [t,t+Ti],  nor  ean  any  user  “guess”  another  user’s  tag  by  any  brute- 
foree  method.  If  eaeh  user  ehooses  tags  randomly,  for  storing  values  and  the  number  of  distinet  tags 
is  suffieiently  large,  the  effeet  is  that  each  user  sees  a  “private  ”  ephemeral  state  store.  Beeause 
a  tag  eollision  is  only  a  problem  if  two  users  use  the  same  tag  within  an  interval  of  length  Ti,  tag 
values  of,  for  example,  64-bits  are  eertainly  within  reason  and  offer  reasonable  proteetion/privaey. 

5.2  Local  Operations  with  Ephemeral  State 

The  set  of  operations  defines  the  eomputations  that  ean  be  performed  using  ephemeral  state.  Eaeh 
operation  is  earried  in  a  speeially  marked  paeket  that  eauses  the  operation  to  be  invoked  at  ESP- 
eapable  routers  during  forwarding.  These  operations  are  analogous  to  the  instruetion  set  of  a 
general-purpose  eomputer:  eaeh  involves  a  small  number  of  operands  and  takes  a  fixed  amounf 
of  fime  fo  eomplefe.  Inferesfing  eompufafions  ean  be  eonsfruefed  by  sequeneing  insfruefions  so  fhaf 
lafer  ones  make  use  of  values  lefl  in  fhe  sfafe  sfore  by  earlier  ones.  The  key  differenees  here  are 
fhaf  (1)  sequeneing  musf  be  aehieved  by  arranging  for  a  sequenee  of  operafion-inifiafing  paekefs  fo 
arrive  af  fhe  roufer  (no  program  eounfer),  and  (2)  eaeh  operafion  ean  only  aeeess  values  plaeed  in 
fhe  sfore  wifhin  fhe  lasf  T;  seeonds. 

Operafions  ean  have  zero  or  more  operands  of  fhe  following  fypes: 

•  a  value  stored  in  fhe  loeal  ephemeral  sfafe  sfore  (i.e.  bound  fo  a  fag  earried  in  fhe  inifiafing 
paekef); 

•  an  “immediafe”  value,  i.e.  one  earried  direefly  in  fhe  paekef; 

•  a  well-known  parameter  value  (for  example,  fhe  value  of  a  MIB  variable). 

Operations  are  enfirely  loeal,  and  eifher  run  fo  eomplefion  or  aborf.  An  operation  fhaf  eomplefes 
sueeessfully  produees  zero  or  more  oufpufs  fhaf  are  eifher  plaeed  in  fhe  paekef  or  bound  fo  a  fag 
in  fhe  ephemeral  sfafe  sfore  or  bofh,  depending  on  fhe  operafion.  Upon  sueeessful  eomplefion, 
depending  on  fhe  values  eompufed  by  fhe  operation  fhe  paekef  fhaf  inifiafed  fhe  operafion  is  eifher 
silenfly  dropped  or  forwarded  foward  ifs  original  desfinafion — possibly  earrying  oufpufs  from  fhe 
eompufafion,  fo  be  used  as  inpufs  af  fhe  nexf  ESP-eapable  node.  Eaeh  operafion  exeeufes  afomieally 
wifh  respeef  fo  fhe  ephemeral  sfafe  sfore;  in  parfieular,  no  binding  ean  expire  during  an  operafion. 

We  envision  fhaf  a  sfandard  sef  of  a  few  dozen  operafions  will  suffiee  for  a  large  elass  of  inferesf- 
ing  eompufafions;  here  we  deseribe  fhree  example  operafions  fhaf  we  have  found  useful  in  building 
aefive  nefwork  serviees. 

COUNTCH 

The  COUNTCH  operafion  eounfs  fhe  number  of  fimes  fhe  operafion  has  been  earried  ouf,  and 
fibers  (bloeks)  paekefs  based  on  fhaf  number.  If  fakes  one  operand,  a  single  lag  earried  in  fhe 
inifiafing  paekef.  If  no  value  is  eurrenfly  bound  to  fhe  lag,  if  is  bound  fo  fhe  value  1  and  fhe 
paekef  is  forwarded.  Olherwise,  fhe  value  bound  fo  fhe  lag  is  inereased  by  1,  and  fhe  inifiafing 
paekef  is  silenfly  disearded.  When  COUNTCH-inilialing  paekefs  earrying  fhe  same  lag  are  senl 
from  a  group  of  hosls  to  a  eommon  desfinafion,  fhe  palhs  followed  by  fhe  paekefs  induee  a 
free.  The  effeel  of  fhe  operation  is  fo  bind  to  fhaf  fag,  af  eaeh  ESP-eapable  roufer,  a  eounl  of 
fhe  number  of  “ehildren”  if  has  in  fhaf  free.  Eaeh  “ehild”  is  an  ESP-eapable  roufer  or  sender. 
This  operafion  is  often  used  as  a  “sef  up”  slep  for  olher  operafions. 
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COLLECT 

The  COLLECT  operation  applies  an  assoeiative  and  commutative  operator  (e.g.,  max,  min, 
sum,  etc.)  to  values  carried  in  COLLECT-initiating  packets.  A  COLLECT-initiating  packet 
contains  two  tags  (say  x  and  y),  two  values  (say  a  and  b),  and  an  operator  code  op  (the 
commutative/associative  operation  to  perform).  COLLECT  expects  one  of  the  tags  x  to  already 
exist  in  the  router’s  ESS.  If  x  does  not  exist  in  the  ESS,  the  operation  aborts.  If  the  other  tag 
y  does  not  exist,  it  is  initialized  and  bound  to  the  value  a  in  the  ESS.  If  tag  y  is  found,  the 
operation  op  is  applied  to  value  of  y  and  value  b,  the  result  is  bound  to  tag  y.  After  each 
successful  operation,  the  value  bound  to  tag  x  is  decremented  by  1.  The  initiating  packet  is 
only  forwarded  if  the  value  bound  to  x  becomes  0;  otherwise  it  is  discarded. 

When  COUNTCH  and  COLLECT  are  sent  by  multicast  receivers  to  the  multicast  sender,  they 
can  help  the  multicast  sender  to  discover  the  number  of  receivers  in  the  multicast  group. 
When  the  op  specified  is  addition,  a  COLLECT-initiating  packet  sent  after  COUNTCH-initiating 
packets  sent  by  all  multicast  receivers  will  reach  the  multicast  sender  containing  the  total 
number  of  the  receivers  in  the  multicast  tree  (assuming  no  losses  occur). 

FMAX 

The  FMAX  operation  tests  whether  a  value  in  the  store  at  a  given  tag  is  greater  than  or  equal 
to  the  value  carried  in  the  packet.  If  so,  the  value  of  a  read-only  system  variable  at  the  node, 
namely  nodeid  (e.g.,  the  node’s  IP  address),  is  stored  in  a  field  in  fhe  ESP  packef.  If  no 
binding  for  fhe  given  lag  exisls,  fhe  packef  is  forwarded  anyway.  This  operalion  is  useful 
for  locafing  nodes  in  fhe  nelwork  lhal  have  desired  properties,  for  example  branch  poinls  of 
mulficasl  frees. 

FMAX  illuslrales  fhe  use  of  well-known  (read-only)  global  variables  in  compulafions — in  fhis 
case,  nodeid.  Each  ESP-capable  node  provides  access  fo  fhis  variable.  Ofher  well-known 
global  variables  may  also  be  useful.  In  fhe  limif,  access  fo  MIB  variables  mighf  be  supporfed. 

Whenever  an  ESP  packef  arrives  af  a  node  (eilher  for  forwarding  or  because  if  is  addressed  fo 
lhaf  node),  if  is  recognized  as  such  and  passed  off  fo  fhe  ESP  module  for  processing. 

Two  forms  of  ESP  are  supported:  dedicated  and  piggybacked.  A  dedicated  ESP  dafagram 
consisfs  of  an  IP  dafagram  whose  payload  confains  fhe  opcode  of  fhe  desired  operalion  along  wilh  ils 
pac kef-borne  operands.  The  IP  header  of  fhe  dafagram  carries  fhe  Router  Alerl  option  (RPC  2113) 
and  fhe  Protocol  Number  of  fhe  ESP  prolocol. 

A  piggybacked  ESP  dafagram  carries  fhe  opcode  and  operands  in  an  IP  oplion,  along  wilh 
a  prolocol  number  and  payload  of  some  regular  applicalion.  Piggybacked  ESP  packels  iniliale 
operations  as  a  side  effecl;  Iheir  advanlage  is  lhal  Ihey  do  nol  add  signficanlly  fo  fhe  bandwidlh 
requiremenls  of  fhe  nelwork. 

Ephemeral  slate  processing  al  each  nelwork  node  is  implemenfed  as  an  adjuncl  fo  fhe  Inlernef 
Protocol  (or  ofher  nelwork-layer  prolocol)  similar  fo  ICMP  or  IGMP.  End  sysfems  musl  implemenl 
fhe  mechanism.  Inferior  nelwork  nodes  should  implemenl  fhe  mechanism,  bul  fhe  service  will 
funclion  correclly  even  if  only  a  subsef  of  inferior  nodes  implemenl  if. 
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5.3  Usage  Scenario 

As  a  simple  example  of  the  use  of  ESP,  we  present  a  method  of  determining  the  identity  of  a  node 
in  the  interseetion  of  two  paths  through  the  network,  if  any. 

Given  are  four  nodes  A,  B,  X,  and  Y.  We  wish  to  find  the  address  of  the  ESP-eapable  node 
elosest  to  B  that  is  on  both  the  paths  A  ^  B  and  X  ^Y.  The  solution  has  two  steps,  and  requires 
the  eooperation  of  A,  X,  and  Y. 

Eirst,  A  transmits  a  COUNTCH-paeket  to  B.  This  has  the  effeet  of  binding  the  value  1  to  the  tag 
z  at  all  ESP-eapable  nodes  along  that  path.  A  short  time  later,  X  transmits  a  FMAX-paeket  to  Y. 
This  paeket  is  sent  with  the  value  0,  the  tag  z,  and  X’s  ID.  If  it  eneounters  a  larger  value  (i.e.  1) 
bound  to  z  at  any  node  along  the  path,  it  eolleets  the  ID  of  that  node  and  earries  it  on  to  Y.  Beeause 
the  eomparison  test  in  FMAX  sueeeeds  on  equality,  the  node  ID  reeeived  at  Y  is  that  of  the  closest 
node  to  Y  on  both  paths. 

As  a  building-bloek  serviee,  ESP  should  be  useful  to  a  variety  of  applieations  and  higher-level 
serviees.  In  other  publieations  we  have  shown  the  following  uses  for  ESP: 

•  In  eombination  with  Eightweight  Proeessing  Modules  that  ean  perform  paeket  duplieation, 
ESP  ean  be  used  to  implement  a  user-eontrolled  multieast  serviee.  That  is,  the  network  pro¬ 
vides  only  unieast  routing  serviees;  ESP  is  used  to  determine  the  best  loeation  in  the  network 
for  duplieation  to  take  plaee  [7] . 

•  Again  in  eombination  with  EWP  modules,  this  time  performing  filtering,  ESP  ean  be  used  to 
perform  eongestion  eontrol  for  multieast  applieations  [12]. 

•  We  have  also  shown  how  ESP  ean  be  used  in  the  eonstruetion  of  a  generie  grouping  serviee, 
whieh  enables  nodes  to  build  more  effieient  overlay  networks. 

5.4  A  Network-Processor-Based  Implementation 

A  key  objeetive  of  high-performanee  (baekbone)  routers  is  the  ability  to  handle  packets  at  wire- 
speeds  on  as  many  interfaces  as  possible.  To  aehieve  this  level  of  performanee,  eurrent  (non¬ 
programmable)  high-speed  routers  perform  paeket  proeessing  in  hardware  using  applieation-speeifie 
integrated  eireuits  (ASICs).  This  approaeh  has  been  highly  sueeessful,  produeing  routers  that  ean 
switeh  millions  of  paekets  per  seeond. 

More  reeently  vendors  have  introdueed  programmable  network  processors  that  ean  perform 
paeket  proeessing  in  parallel.  To  evaluate  ESP’s  ability  to  seale  to  highend  routers,  we  used  the 
Intel  IXP  1200  network  proeessor^  as  our  development  platform.  The  IXP  1200  is  designed  for 
high-performanee,  deep  paeket  proeessing,  and  has  general  eharaeteristies  similar  to  those  of  other 
network  proeessors  sueh  as  programmability,  multiple  proeessing  engines,  and  a  multi-level  mem¬ 
ory  hierarehy.  Consequently,  we  expeet  that  insights  derived  from  our  experienee  with  the  IXP  1200 
ean  be  applied  to  other  network  proeessor  platforms  as  well. 

5.4.1  Mapping  ESP  to  the  IXP  1200 

The  Intel  IXP  1200  network  proeessor,  illustrated  in  Eigure  1 1,  is  designed  to  faeilitate  the  deploy¬ 
ment  of  new  value-added  serviees  simply  by  ehanging  the  software  running  on  the  hardware.  To 

Intel  Corporation  provided  additional  support  for  this  work. 
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PCI  Bus 


IX  Bus 


Figure  11:  IXP  1200  architecture 


achieve  line-rate  routing/switching,  the  IXP  1200  supports  parallel  packet  processing  on  a  set  of 
six  processors  called  micro-engines.  The  six  on-chip  micro-engines  each  supports  four  hardware 
threads.  Each  of  the  micro-engines  is  individually  programmable  and  supports  a  set  of  hardware 
instructions  specifically  designed  for  packet  processing.  Like  other  network  processors,  the  IXP 
1200  uses  a  multi-level  memory  hierarchy  consisting  of  registers,  SRAM,  and  SDRAM.  Registers 
offer  the  fastest  performance,  with  SRAM  and  SDRAM  performance  being  significantly  slower; 
SDRAM  being  worse  than  SRAM.  However,  the  SDRAM  is  much  larger  than  the  SRAM  which  is 
much  larger  than  the  register  set. 

To  achieve  the  desired  level  of  performance,  we  defined  a  set  of  design  goals  for  our  IXP  1200 
implementation: 

•  Maximize  Parallelism/Minimize  Synchronization  Overhead. 

•  Minimize  Memory  Accesses.  Map  data  structures  to  storage  so  that  ESS  information  can  be 
retrieved  with  as  few  accesses  as  possible. 

•  Maximize  ESS  Capacity.  Use  memory  efficiently. 

•  Maintain  ESP  Semantics.  Maintain  the  ordering  and  sharing  constraints  described  above. 

•  Operate  Continuously. 

To  achieve  parallelism,  we  distribute  the  ESP  processing  across  the  set  of  IXP  micro-engines. 
Micro-engine  0  serves  as  the  ingress  processor  and  packet  dispatcher,  micro-engine  5  collects  all 
outgoing  packets  and  enqueues  them  for  transmission,  and  micro-engines  1  to  4  handle  all  ESP 
processing. 

Because  we  want  the  ESS  to  be  as  large  as  possible,  we  store  the  ESS  data  itself  (tags  and 
values)  in  SDRAM.  Although  SDRAM  has  the  highest  read/write  latency  of  any  of  the  IXP  1200’s 
storage  areas,  it  is  only  a  bit  higher  than  the  latency  of  the  SRAM,  and  it  offers  significantly  larger 
capacity.  Smaller  auxiliary  data  structures  (hash  tables)  are  stored  in  SRAM  for  speed. 
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ESP  requires  at  least  one  ESS  per  network  interfaee,  but  there  are  advantages  to  having  more 
ESS’s  per  interfaee.  Eirst,  partitioning  eomputations  into  different  ESS’s  reduees  the  number  of 
tags  per  ESS,  thereby  redueing  the  probability  of  tag  eollisions.  Seeond,  reelamation  of  expired 
data  requires  synehronization  between  the  eleaning  thread  and  the  paeket-proeessing  threads;  on 
the  IXP1200,  synehronization  aeross  miero-engines  is  signifieantly  more  expensive  than  between 
threads  on  the  same  miero-engine.  Therefore  in  our  design,  eaeh  miero-engine  (as  opposed  to  eaeh 
interfaee)  has  its  own  ESS,  used  exelusively  for  paekets  proeessed  by  that  miero-engine.  To  ensure 
that  paekets  are  proeessed  in  the  eorreet  ESS  eontext,  we  use  a  hash  of  the  eomputation  ID  to  assign 
paekets  to  ESSs. 

Although  parallelizing  the  proeessing  aeross  multiple  miero-engines  results  in  improved  perfor- 
manee,  we  took  it  a  step  farther,  by  using  multiple  threads  per  miero-engine.  Although  all  threads 
exeeute  on  the  same  CPU,  the  ability  to  switeh  to  a  different  thread  while  bloeked  waiting  for  a 
memory  aeeess  to  eomplete  (say  to  SRAM  or  SDRAM),  allows  us  to  mask  the  high  lateney  as- 
soeiated  with  these  operations.  However,  eoneurrent  proeessing  by  multiple  threads  raises  several 
interesting  issues.  Eirst,  to  ensure  that  paekets  from  the  same  eomputation  are  not  proeessed  eoneur- 
rently  by  different  threads,  the  paeket  demultiplexing  eode  dispatehes  all  paekets  having  the  same 
eomputation  id  to  the  same  one  of  four  input  queues  (not  just  the  same  miero-engine).  Eaeh  thread 
ean  serviee  any  input  queue;  however,  we  use  loeal  thread  synehronization  teehniques  to  ensure 
that  no  two  threads  work  on  the  same  queue  at  the  same  time.  Seeond,  given  the  ehallenges  of 
synehronizing  multiple  eoneurrent  threads,  it  might  make  sense  to  further  subdivide  the  ESS  into 
subESS’s,  with  one  subESS  per  thread.  The  downside  of  this  approaeh  is  that  memory  ean  beeome 
fragmented,  wasting  valuable  ESS  spaee.  Although  it  would  simplify  synehronization,  we  deeided 
to  stiek  with  a  single  ESS  per  miero-engine  (shared  by  all  threads). 

To  support  eontinuous  operation  of  the  router,  we  used  one  thread  on  eaeh  miero-engine  as  a 
cleaner  thread,  reelaiming  expired  (tag,value)  pairs  from  the  ESS.  Beeause  the  eleaner  examines  and 
modifies  the  state  of  the  ESS,  there  is  potential  for  interferenee  between  it  and  a  normal  ESP  instrue- 
tion  being  exeeuted  eoneurrently  by  another  thread.  To  address  this  issue  we  designed  an  effieient 
synehronization  method  that  allows  ESS  aeeesses  by  both  the  eleaner  and  instruetion-proeessing 
threads  to  be  interleaved  with  minimal  synehronization. 

5.4.2  IXP  Performance  Results 

To  evaluate  the  performanee,  we  measured  the  throughput  aehievable  with  multiple  threads.  All 
threads  exeeuted  on  the  same  miero-engine,  so  there  was  no  direet  inerease  in  parallelism;  however, 
the  use  of  multiple  threads  resulted  in  overlapping  SDRAM  aeeesses,  hiding  (at  least  some  of)  the 
lateney  of  reading  and  writing.  To  generate  enough  traffie  to  keep  the  threads  busy,  we  ran  eight  PCs 
simultaneously  transmitted  minimum-sized  paekets  as  fast  as  possible.  Eaeh  paeket  earried  a  count- 
with-threshold  instruetion;  50%  of  the  paekets  ereated  a  new  tag,  and  50%  simply  ineremented  an 
existing  tag.  Beeause  paekets  were  minimum  sized  (requiring  an  inter-paeket  gap),  the  total  offered 
load  we  were  able  to  generate  was  (76.25  Mbps  x  4  lines)  =  305  Mbps. 

Table  2  shows  the  throughput,  both  in  paekets  per  seeond  and  megabits  per  seeond,  for  1,  2,  and 
3  threads.  The  fourth  thread  is  the  eleaner.  The  “No  Cleaner-assist”  table  shows  performanee  when 
the  eleaner  does  nothing  but  elean.  The  “With  Cleaner-assist”  table  shows  performanee  when  the 
eleaner  uses  its  spare  eyeles  to  assist  with  paeket  proeessing.  Clearly,  at  lower  loads,  the  eleaner 
has  more  eyeles  to  assist  with  paeket  proeessing  than  at  high  loads,  and  the  throughput  with  one 
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No  Cleaner-assist 

No.  Threads 

Throughput 

1 

234  Kpps  =  120  Mbps 

2 

420  Kpps  =  215  Mbps 

3 

547  Kpps  =  280  Mbps 

With  Cleaner-assist 

No.  Threads 

Throughput 

1 

212  Kpps  =  108  Mbps 

2 

410  Kpps  =  210  Mbps 

3 

507  Kpps  =  260  Mbps 

4 

564  Kpps  =  290  Mbps 

Table  2:  Throughput  of  a  single  miero-engine,  with  and  without  eleaner-assist. 


paeket-proeessing  thread  plus  the  eleaner  assist  is  eomparable  to  the  throughput  with  two  paeket- 
processing  threads.  Reeall  that  the  maximum  aehievable  throughput  (with  minimum-size  paekets) 
is  305  Mbps  due  to  Ethernet  limitations.  With  three  threads  (plus  the  eleaner  in  its  spare  time) 
processing  packets,  the  IXP  comes  within  5%  of  the  full  305  Mbps.  In  fact,  when  we  changed  the 
mix  of  instructions  so  that  all  packets  carry  the  same  tag,  the  IXP  operated  at  the  optimal  305  Mbps 
rate.  Note  that  this  level  of  performance  is  achieved  with  only  a  single  micro-engine  doing  packet 
processing. 

5.5  A  Modular  Software  ESP  Implementation 

Not  all  routers  emphasize  performance  and  scalability.  For  example,  “broadband  router/gateway” 
products  for  the  home  or  office  support  small-scale  systems,  but  more  advanced  functionality  (e.g., 
firewall  protection).  Similar  goals  apply  for  military  network  systems  designed  for  quick  field  de¬ 
ployment.  The  design  considerations  for  such  routers  are  quite  different  from  those  of  the  high-end 
routers;  instead  of  performance  and  parallelization,  these  systems  focus  on  robustness,  flexibility, 
and  minimizing  cost.  To  achieve  this  flexibility,  many  of  these  routers  employ  standard  processors 
executing  conventional  operating  systems  (like  Linux),  modified  to  support  the  router  features  the 
vendor  desires. 

Click  is  a  fine-grained  modular  soflware  approach  for  implementing  routers  with  advanced  and 
evolving  packet-processing  features;  as  such,  it  is  ideal  for  our  purposes.  The  basic  paradigm  of 
click  is  to  decompose  router  functionality  into  small  functions  that  can  be  performed  more  or  less 
independently,  and  encapsulate  them  in  individual  modules  called  elements.  Elements  are  organized 
into  a  processing  graph. 

To  incorporate  ESP  processing  in  routers  such  as  click,  we  divided  the  ESP  functionality  into 
modules  (click  elements)  and  incorporated  them  into  the  click’s  packet  flow  graphs.  One  of  the 
main  problems  we  faced  was  how  to  modularize  ESP.  The  click  philosophy  is  for  modules  to  be 
small  and  simple;  this  would  suggest  separate  modules  to  validate  the  ESP  header,  retrieve  val¬ 
ues  from  the  ESS,  execute  ESP  instructions,  and  classify  packets.  However,  a  highly-decomposed 
implementation  raises  issues  about  performance  and  sharing  of  data  between  modules. 

We  implemented  most  of  the  ESP  functionality,  including  instruction  execution  and  state  store 
management,  in  a  single  module  called  ESPExec.  Some  of  the  benefits  of  modularity  are  given 
up  this  way,  but  probably  not  too  many.  For  example,  it  is  not  clear  that  there  is  much  need  for 
re-use  of  functional  modules  within  ESP.  In  addition,  the  standard  composition  technique  in  click 
has  well-known  performance  costs,  and  an  integrated  implementation  avoids  them. 

To  evaluate  the  overhead  of  our  click  implementation,  we  compared  against  the  cost  of  basic  IP 
processing  in  click.  The  click  authors  report  the  time  to  process  an  IP  packet  is  roughly  2900  ns  on 
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a  PHI  running  at  700  MHz  for  a  total  forwarding  rate  of  345  Kpps.  To  baseline  our  system,  a  P4 
running  at  1.8GHz,  we  reran  the  same  experiment  as  the  eliek  authors,  and  found  the  time  to  proeess 
an  IP  paeket  to  be  roughly  1300  ns  for  a  total  forwarding  rate  of  765  Kpps.  Given  this  base-level 
performanee,  we  are  able  to  gauge  the  overhead  required  by  ESP. 


count 

compare 

collect 

rcollect 

660  ns 

702  ns 

1064  ns 

1525  ns 

Table  3:  Added  eost  of  ESP  proeessing. 

The  additional  ESP  proeessing  eost  depends  on  the  ESP  instruetion.  The  count-with-threshold 
and  compare  instruetions  are  only  55%  more  eostly  than  standard  IP  proeessing.  The  ESP  collect 
instuetion,  on  the  other  hand,  involves  more  tags  and  thus  inereases  the  overhead  by  81%.  Eor  the 
more  eomplex  rcollect,  the  overhead  is  about  116%.  Although  the  overhead  is  substantial,  the  worst 
ease  throughput  (with  minimum  sized  paekets)  is  still  354  Kpps  or  181  Mbps,  whieh  is  suffieient  to 
keep  all  lines  on  a  home  router  busy. 

6  Lightweight  Processing  Modules 

EWP  is  a  paradigm  for  providing  enhaneed  serviees  under  user  eontrol.  It  is  designed  to  leverage 
the  eapability  provided  by  ESP,  and  is  based  on  two  ideas: 

•  Certain  common  and  very  simple  functionality  is  useful  for  implementing  a  variety  of  packet¬ 
processing  services.  Eor  example,  the  ability  to  extract  packets  matching  a  particular  pattern 
and  duplicate,  redirect,  or  drop  them  is  a  useful  building  block  from  which  many  services  can 
be  built.  (This  observation  also  motivates  the  design  of  many  “network  processor”  devices, 
such  as  the  Intel  IXP1200.) 

•  It  is  simpler  for  an  application  to  communicate  directly  with  a  node  in  the  middle  of  the 
network  to  invoke  enhanced  functionality  than  it  is  to  have  the  invocation  request  propagate 
from  node  to  node,  across  all  intervening  domains,  subject  to  each  node’s  policies.  The  “re¬ 
lay”  model  is  appropriate  when  the  functionality  must  be  installed  at  a  large  number  of  nodes, 
as  in  concast.  However,  as  we  saw  earlier,  it  does  require  that  the  application  trust  intermedi¬ 
aries  to  convey  the  request  and  any  response  correctly.  When  functionality  is  required  at  one 
or  a  small  number  of  nodes,  it  is  better  for  the  application  to  negotiate  directly  with  the  node 
or  nodes  concerned. 

An  additional  benefit  of  this  paradigm  is  a  simplified  paymenf  model:  fhe  requesting  appli¬ 
cation  can  negofiafe  direcfly  wifh  fhe  providing  node  regarding  paymenf.  This  allows  for  fhe 
possibilify  of  domain-specific  charging  arrangemenfs,  whereas  in  fhe  requesf-relay  model,  fhe 
requesf  musf  already  confain  adequate  billing  informalion  for  whafever  node/domain  ends  up 
providing  fhe  service. 

As  we  have  seen,  ESP  allows  end  systems  fo  identify  regions  in  fhe  nefwork  fhaf  are  sfrafegically 
useful  for  an  applicafion — a  pafh  infersecfion  poinf,  a  congesfed  link,  or  fhe  confluence  of  mulfiple 
streams. 
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6.1  LWP  Specification 

LWPs  are  predefined  proeessing  modules  supported  by  network  routers  that  ean  be  aetivated  by  end 
systems  via  eontrol  messages.  Eaeh  proeessing  module  applies  some  funetion  (sueh  as  duplieation) 
to  paekets  passing  through  the  router  that  mateh  a  speeified  pattern.  The  funetion  is  eontrolled 
through  parameters  speeified  at  setup  time.  Only  an  authorized  end  user  ean  instantiate  an  LWP  for 
paekets  assoeiated  with  the  end  system.  Eaeh  LWP  has  an  assoeiated  end  system  that  is  responsible 
for  its  existenee  (e.g.  that  refreshes  the  LWP  state).  This  model  simplifies  the  seeurity  and  resouree 
proteetion  of  the  LWP  system.  We  expeet  that  a  small  set  of  these  parameterized  modules,  whieh 
may  be  sequeneed  for  paeket  proeessing,  will  suffiee  for  a  variety  of  network  serviees. 

Eaeh  proeessing  module  is  defined  by  the  following  four  eomponents: 

•  Module  ID:  the  funetion  to  exeeute  at  the  router  (or  aetual  eode). 

•  Classifier:  a  paeket  filter  that  identifies  whieh  paekets  this  module  should  intereept,  for  ex¬ 
ample  a  value  indieating  the  multieast  group  ID  to  intereept  for  duplieation. 

•  Parameters:  a  set  of  eonfiguration  parameters  that  eontrol  the  eode’s  exeeution.  Lor  example, 
an  instantiation  parameter  might  speeify  the  unieast  address  where  paekets  output  by  this 
module  are  to  be  sent. 

•  Timeout:  a  timeout  value  indieating  when  the  module  should  be  automatieally  removed  from 
the  router.  There  may  be  a  system-wide  maximum  timeout  value  imposed  on  all  modules. 
Refresh  messages  are  used  to  extend  a  module’s  lifetime. 

Lightweight  proeessing  modules  operate  by  identifying  paekets  that  mateh  a  partieular  elassifier, 
applying  the  speeified  proeessing  to  those  paekets,  and  (possibly)  forwarding  the  result(s)  using 
standard  unieast  routing.  A  proeessing  module  may  be  instantiated  multiple  times  on  a  router,  where 
eaeh  instantiation  differs  in  either  the  elassifier,  parameters,  or  timeout.  Modules  ean  be  instantiated, 
terminated,  and  refreshed  via  the  eontrol  meehanism.  The  refresh  operation  may  dynamieally  alter 
the  parameters  used  by  a  module.  Note  that  a  paeket  may  mateh  multiple  lightweight  proeessing 
modules  at  a  router,  and  thus  be  proeessed  multiple  times. 

An  example  of  LWP  is  the  dup  { )  module,  whieh  replieates  any  paeket  matehing  its  elassifier 
filter  and  forwards  the  duplieate  to  a  unieast  address  speeified  at  module  instantiation  time  as  a 
eonfiguration  parameter.  The  original  paeket  is  simply  forwarded  normally  to  its  original  (unieast) 
destination. 

6.2  Usage  Scenarios 

Using  this  and  other  LWP  modules  together  with  ESP,  we  have  shown  how  to  implement  multieast 
based  on  unieast  routing  [7].  In  our  seheme,  multieast  paekets  are  direeted  to  a  unieast  address  (i.e., 
the  destination  address  is  always  one  of  the  reeeivers),  but  earry  a  multicast  group  ID  in  an  option 
of  IPv4  header  (or  an  extension  header  of  an  IPv6  paeket).  The  elassifier  of  the  dup  { )  funetion 
speeifies  this  multieast  group  ID.  The  funetion  makes  a  eopy  of  eaeh  matehing  paeket,  replaees  the 
eopy’s  original  IP  destination  address  with  its  own  eonfigured  destination  address,  and  forwards 
both  the  original  and  the  eopy  normally.  Thus,  dup  { )  funetions  eorrespond  to  “braneh  points” 
in  a  multieast  delivery  tree.  By  plaeing  the  dup  { )  funetions  strategieally,  end  systems  ean  ereate 
applieation-speeifie  multieast  delivery  trees.  ESP  is  used  to  determine  where  to  plaee  the  dup  { ) 
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functions.  We  have  developed  two  schemes  for  creating  application-level  multicast  trees:  one  using 
a  centralized  approach  (sender-controlled)  and  a  second,  receiver  controlled  approach  [7]. 

In  the  receiver-controlled  approach,  the  receivers  collaborate  using  ESP  to  identify  the  branch 
points  in  the  existing  multicast  tree.  A  new  receiver  then  adds  a  dup()  function  to  an  existing 
branch  point.  Periodically  the  sender  transmits  tree  optimization  messages  that  will  result  in  the 
new  receiver’s  dup()  function  being  moved  to  the  optimal  location  in  the  tree.  An  example  multicast 
tree  built  in  this  manner  is  shown  in  Fig  12. 


This  ESP/LWP  multicast  tree  structure  can  be  easily  extended  to  support  layered  multicast. 
Eayered  multicast  has  been  proposed  for  video  transmission,  where  the  sender  encodes  the  multicast 
content  into  different  layers  and  then  sends  them  on  different  multicast  groups.  Usually  each  layer 
improves  the  video  quality  incrementally.  For  example,  receiving  the  base  layer  gives  the  lowest 
quality  video  and  each  additional  layer  improves  the  video  quality.  In  traditional  layered  multicast, 
receivers  join  as  many  multicast  groups  as  possible  to  receive  the  video  streams  at  the  fastest  rate 
supported  by  the  receiver’s  connection. 

In  our  EWP  multicast  implementation,  layers  are  identified  not  by  different  multicast  groups, 
but  by  header  or  IP  option  on  which  the  dup()  function  classifies;  fhus  a  single  dup()  funclion  can 
duplicafe  mulfiple  layers  of  dafa.  Using  drop()  funclions  in  addifion  fo  dup(),  a  receiver  in  a  layered 
mulficasf  session  can  dynamically  adjusf  fhe  arriving  dafa  rale.  The  mullicasl  sender  and  receivers 
also  monilor  fhe  congestion  on  fhe  mullicasl  palh  using  ESP  compulation  in  a  collaborative  effort 
We  have  designed  such  a  layered  mullicasl  approach  called  Congestion-Aware  Eayered  Mullicasl 
(CAEM).  Compared  lo  Iradilional  layered  mullicasl  implemenlalions,  CAEM  receivers  can  more 
accurately  deled  congestion  on  Ihe  mullicasl  palh,  and  read  lo  congestion  in  a  more  timely  manner. 
We  have  shown  lhal  CAEM  also  provides  more  slable  receiving  qualify  lhan  a  Iradilional  approach. 
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The  details  of  our  approaeh  along  with  results  are  presented  in  [12]. 


7  Speccast:  Generalized  Routing  and  Forwarding 

The  speeeast  serviee  is  defined  as  follows.  Let  N  denote  a  set  of  nodes,  and  let  P  be  a  set  of 
predieates  on  nodes,  i.e.  functions  from  N  to  {true, false}.  Each  packet  m  carries  a  predicate 
m.dest  G  P  called  its  destination  predicate.  The  predicate  specifies  fhe  sef  of  nodes  fo  which  m 
is  fo  be  delivered.  Thai  is,  fhe  nodes  of  fhe  nelwork  conspire  lo  deliver  m  lo  all  nodes  n  such  lhal 
m.dest(n)  —  true. 

The  speccasl  service  is  best-effort',  if  delivers  fhe  message  m  lo  nodes  salisfying  m.dest,  pro¬ 
vided  cerlain  condilions  are  mel  (e.g.,  fhe  nelwork  is  nol  overloaded).  Moreover,  each  packel  is 
forwarded  and  delivered  independenlly  of  all  olhers,  i.e.  speccasl  is  a  datagram  service. 

This  problem  slalemenl  is  very  general.  Il  says  nolhing  aboul  Ihe  nalure  of  Ihe  predicates  used 
lo  specify  deslinalions,  nor  does  il  assume  anylhing  aboul  which  nodes  salisfy  whal  properlies.  In 
particular,  no  relationship  is  assumed  belween  Ihe  predicates  satisfied  by  a  node  and  lhal  node’s 
location  in  Ihe  topology,  and  no  underlying  routing  or  addressing  mechanism  is  assumed.  Defining 
a  solution  to  Ihis  problem  involves  specifying  (i)  Ihe  information  carried  in  each  packel  in  addi¬ 
tion  to  Ihe  destination  predicate;  (ii)  Ihe  information  stored  al  each  node,  i.e.  Ihe  “forwarding  dala 
slruclure”;  (iii)  Ihe  algorilhm  used  by  nodes  to  originale/forward  packels,  which  lakes  as  inpuls 
Ihe  forwarding  dala  slruclure  and  Ihe  information  from  Ihe  packel,  and  relurns  Ihe  sel  of  channels 
on  which  Ihe  packel  should  be  Iransmilled;  and  (iv)  dislribuled  algorilhms  for  initializing  Ihe  for¬ 
warding  dala  slruclures  al  each  node  and  updating  Ihem  across  changes  in  nelwork  topology  (and 
possibly  in  predicate  sel,  i.e.  which  nodes  salsify  which  predicates).  In  general,  Ihe  delails  of  Ihese 
algorilhms  will  depend  on  Ihe  properties  of  P,  and  Iheir  efficiency  will  depend  on  which  nodes  in 
Ihe  topology  satisfy  which  predicates. 

Depending  upon  Ihe  properties  of  P  and  which  nodes  satisfy  whal  predicates,  Ihe  speccasl  ser¬ 
vice  abslraclion  can  subsume  many  Iradilional  and  emerging  nelwork-layer  addressing/routing  ser¬ 
vices.  For  example,  unicast  service  is  supported  if  P  conlains  a  point  predicate  for  each  node  to 
which  messages  need  to  be  delivered.  (A  poinl  predicate  is  satisfied  by  exaclly  one  node  and  Ihus 
can  be  used  as  an  identifier.)  Classical  (any-source)  multicast  service  can  be  supported  by  having  a 
predicate  in  P  for  each  mullicasl  group  address,  which  is  satisfied  by  exaclly  Ihe  nodes  belonging 
to  lhal  mullicasl  group.  A  publish  and  subscribe  service  can  be  provided  if  Ihere  is  a  predicate  in 
P  for  each  published  elemenl,  which  is  satisfied  by  all  nodes  subscribing  to  receive  information 
corresponding  to  lhal  elemenl.  In  shorl,  Ihe  flexibilily  of  predicates  allows  Ihe  speccasl  service  to 
be  used  in  a  wide  variely  of  ways.  In  addition,  Ihis  generalized  service  is  useful  for  sludying  Ihe  re¬ 
lationships  among  address  slruclure,  localily,  and  routing/forwarding  efficiency.  An  understanding 
of  Ihese  relationships  is  crucial  for  Ihe  design  of  fulure  nelwork  archileclures. 

7.1  Solution  Evaluation  Metrics 

To  compare  solutions  to  Ihe  speccasl  problem,  we  define  several  measures  of  Ihe  cosl  of  delivering 
a  packel  carrying  a  given  destination  predicate  p  from  a  given  source  node  n  to  all  nodes  lhal  satisfy 

p: 

•  Network  load,  i.e.  Ihe  number  of  hops  (links)  over  which  a  message  is  Iransmilled.  Note 
Ihis  includes  all  links  over  which  Ihe  message  is  forwarded,  even  if  Ihe  link  does  nol  lead  to 
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any  node  that  satisfies  p.  We  define  the  network  load  ratio  to  be  the  ratio  of  the  total  number 
of  edges  erossed  by  a  message  versus  the  number  of  links  in  the  shortest-path  tree  from  the 
souree  n  to  all  nodes  satisfying  p. 

•  Delay.  We  define  the  delay  cost  of  a  routing  solution  to  be  the  sum,  over  all  nodes  d  satisfying 
p,  of  the  number  of  edges  erossed  on  the  way  from  n  to  d,  when  that  solution  is  used.  We 
define  delay  stretch  to  be  the  ratio  of  the  algorithm’s  measured  delay  eost  versus  the  minimum 
possible  delay  eost. 

•  Network  State.  We  measure  the  state  cost  of  a  solution  as  the  amount  of  information  stored 
at  all  nodes  eombined.  In  general  this  is  affeeted  by  the  size  of  N  and  the  strueture  of  P. 

•  Forwarding-time  Computational  Cost.  We  measure  the  total  computational  costs  as  the 
sum  aeross  all  nodes  of  the  forwarding  proeessing  eost  (i.e.,  the  number  of  eomputation  steps 
needed  to  deeide  where/how  to  forward  the  message).  This  eost  depends  heavily  on  the 
properties  of  P. 

We  have  investigated  two  different  solutions  to  the  speeeast  problem.  We  deseribe  eaeh  one  in  turn. 

7.2  A  Layered  Solution 

We  have  developed  and  evaluated  a  speeeast  solution  for  a  partieular  predieate  language:  positive 
eombinations  of  atomie  propositions  in  disjunetive  normal  form.  Examples  of  possible  destination 
predieates  in  P  for  this  language  inelude  b,  a  f\b  f\  c,  aW  d,  and  [a  A  b)  y  c.  Here  a,  b  and  c  are 
atomie  propositions,  eaeh  of  whieh  is  either  satisfied  or  not  by  eaeh  node;  sueh  atomie  propositions 
ean  represent  attributes  sueh  as  “command  node”,  “altitude  >  5000”,  etc.  Negation  is  not  included, 
but  could  be  added  fairly  straightforwardly  at  the  cost  of  additional  state. 

The  basic  idea  of  our  approach  for  this  language  is  to  decompose  the  problem  into  two  sub¬ 
problems,  and  solve  each  subproblem  with  a  separate  layer.  The  first,  lower-level  subproblem  is 
to  deliver  a  packet  to  all  nodes  satisfying  a  single  atomic  proposition  b.  We  refer  to  the  solution  to 
this  subproblem  as  the  “base  layer”.  The  second,  higher-level  subproblem  is  to  use  the  base  layer 
to  deliver  a  packet  to  the  set  of  nodes  satisfying  any  given  disjunction  of  conjunctions  (i.e.  any 
boolean  expression  in  disjunctive  normal  form)  while  minimizing  delivery  to  nodes  that  do  not  sat¬ 
isfy  the  predicate.  We  refer  to  the  solution  to  the  second  subproblem  as  the  “composition  layer”; 
the  composition  layer  runs  on  top  of  the  base  layer. 

7.2.1  Base  Layer:  Deliver  to  Atomic  Propositions 

The  problem  of  delivery  to  nodes  satisfying  each  atomic  proposition  can  be  solved  using  an  any- 
source  multicast  service  (i.e.  classical  IP  multicast),  simply  by  using  a  multicast  group  address  per 
atomic  proposition  and  having  each  node  that  satisfies  an  atomic  proposition  join  the  corresponding 
multicast  group.  Unfortunately,  for  efficient  delivery,  both  the  base  layer  and  the  composition  layer 
need  to  process  the  packet  at  each  hop.  Because  this  is  not  possible  using  a  legacy  multicast  service, 
we  developed  our  own  base  layer,  which  constructs  a  spanning  tree  per  atomic  proposition  6;  this 
spanning  tree  contains  all  the  nodes  that  satisfy  b.  In  general  this  tree  will  also  include  nodes  that 
do  not  satisfy  b,  i.e.  it  is  a  Steiner  tree. 
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Every  node  in  the  network  maintains  a  base  layer  forwarding  table  containing  one  entry  per 
atomic  proposition.  If  node  n  is  part  of  the  tree  for  attribute  b,  the  table  entry  for  b  indicates  n’s 
neighbors  in  the  tree.  If  n  is  not  part  of  the  tree  for  b,  the  entry  for  b  indicates  the  neighbor  that  is 
“closest”  to  that  tree  among  all  of  n’s  neighbors. 

When  a  packet  needs  to  be  forwarded  to  nodes  satisfying  a,  the  node  looks  up  the  entry  for  a 
and  transmits  a  copy  of  the  packet  over  all  associated  interfaces,  except  the  one  on  which  the  packet 
arrived,  if  any.  Thus,  once  a  packet  reaches  the  tree  for  atomic  proposition  a,  it  is  forwarded  to  all 
nodes  satisfying  o.  The  use  of  a  shared  tree  minimizes  overall  network  load  and  network  state  is 
reduced,  possibly  at  the  cost  of  increased  delay. 

The  forwarding  table  information  for  the  base  layer  can  be  established  using  a  simple  distance- 
vector-type  protocol  in  which  each  node  announces  to  its  neighbors  the  information  it  knows  about 
every  attribute  tree:  the  attribute,  the  distance  from  the  node  to  the  tree  (0  if  the  node  satisfies  the 
attribute,  infinity  if  it  does  not  know  of  a  path  to  the  tree),  and  a  tree  ID  (which  can  be  chosen 
randomly).  Initially  each  node  considers  itself  the  only  member  of  the  tree  for  exactly  those  atomic 
propositions  that  it  satisfies.  When  a  node  learns  of  another  tree  for  the  same  atomic  proposition 
with  a  lower  ID,  it  “joins”  that  tree;  the  result  is  that  eventually  the  two  trees  merge  into  one  (for 
details,  refer  to  [21]).  Nodes  on  the  tree  also  include  an  estimate  of  the  tree  size;  this  is  used  in 
the  composition  layer.  This  process  does  not  produce  a  minimal  tree;  however,  the  problem  of 
constructing  a  minimum  Steiner  tree  is  well-known  to  be  computationally  hard. 

7.2.2  Composition  Layer 

To  reach  the  set  of  nodes  satisfying  a  boolean  expression  in  disjunctive  normal  form — ^recall  that 
any  arbitrary  boolean  expression  can  be  expressed  in  disjunctive  normal  form — the  composition 
layer  uses  the  base  layer  in  a  two-step  process.  First,  the  originating  node  selects  a  single  atomic 
proposition  from  each  conjunction  in  the  expression.  Then  a  copy  of  the  packet  is  forwarded  to 
all  nodes  that  satisfy  each  such  selected  atomic  proposition,  using  the  base  layer.  This  process  of 
“weakening”  the  destination  predicate  results  in  some  nodes  receiving  the  packet  that  do  not  satisfy 
its  original  destination  predicate,  but  every  node  that  satisfies  the  original  predicate  satisfies  fhe 
weaker  version  and  will  receive  the  packet.  Thus  we  are  trading  some  additional  network  load  for  a 
reduction  in  the  amount  of  forwarding  state  needed. 

For  example,  the  predicate  D  —  {a  A  b)  y  c  might  be  weakened  to  o  V  c  or  to  D'  =  6  V  c.  The 
services  of  the  base  layer  are  then  used  to  send  the  packet  to  nodes  satisfying  the  weaker  predicate, 
i.e.  all  nodes  satisfying  b  and  all  nodes  satisfying  c.  Forwarding  by  the  base  layer  is  based  on  D', 
although  the  original  predicate  D  remains  in  the  packet.  At  each  node  m,  the  predicate  D  is  checked 
to  see  whether  it  is  satisfied  by  m;  if  so,  the  packet  is  delivered  to  the  applications  layers  of  the  node. 

To  decide  which  atomic  proposition  to  select  from  each  conjunction,  we  use  a  heuristic,  based 
on  the  combination  of  tree  size  and  distance  from  the  originating  node  to  the  tree;  both  are  approxi¬ 
mately  proportional  to  the  network  load  (number  of  links  traversed)  that  will  result  from  delivering 
the  packet  via  that  tree.  Therefore  we  choose  the  atomic  proposition  from  each  conjunction  that 
minimizes  the  sum  of  these  values.  After  this  weakening  step,  the  resulting  list  of  selected  atomic 
propositions  is  placed  in  the  packet  along  with  the  original  destination  predicate,  and  the  packet  is 
forwarded  to  each,  using  the  base  layer.  (Recall  that  each  node  has  a  forwarding  table  entry  for  each 
attribute,  regardless  of  whether  it  satisfies  that  attribute  or  not.) 

As  an  example,  consider  the  predicate  D  —  (a  A  6)  V  (c  A  d).  Suppose  that  a  and  c  are  the 
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Figure  13:  Packet  forwarding  example 


atomic  propositions  with  the  smallest  sum  of  distance-to-tree  and  tree  size.  The  packet  is  forwarded 
with  a  header  containing  a  list  of  disjunctions,  each  consisting  of  a  conjunction  and  an  indication  of 
the  atomic  proposition  selected  from  that  conjunction,  plus  a  pending  bit  for  that  atomic  proposition 
(indicated  in  the  figure  by  the  number  in  parentheses): 


a  Ab 

o(l) 

c  Ad 

c(l) 

The  pending  bit  for  a  indicates  whether  the  packet  still  needs  to  reach  the  tree  for  a. 

When  forwarding  a  packet,  the  composition  layer  at  node  n  carries  out  the  following  steps. 

1.  For  each  selected  atomic  proposition  a  such  that  n  is  part  of  the  tree  for  o:  forward  the  packet 
out  all  interfaces  belonging  to  that  tree  (except  the  one  on  which  the  packet  arrived),  clearing 
the  pending  bit  for  a  in  the  process. 

2.  For  each  selected  atomic  proposition  b  such  that  n  is  not  part  of  the  tree  for  b  and  the  pending 
bit  is  set:  add  the  interface  toward  6’s  tree  to  the  list  of  outgoing  interfaces  for  the  packet. 

3.  For  each  interface  in  the  list  created  above,  forward  a  copy  of  the  packet  on  that  interface, 
clearing  the  pending  bits  for  each  atomic  proposition  whose  tree  does  not  lie  in  that  direction. 

These  steps  ensure  that  the  packet  follows  the  shortest  path  to  the  tree  for  each  selected  atomic 
proposition.  They  also  allow  a  single  copy  of  a  packet  to  follow  a  single  path  toward  multiple  trees 
until  the  paths  diverge. 

Note  that  the  first  step  above  results  in  packets  traversing  the  union  of  trees  that  intersect.  That 
is,  if  a  packet  is  traversing  one  spanning  tree  and  reaches  a  node  that  also  belongs  to  the  tree  of  one 
of  the  other  selected  atomic  propositions,  the  composition  layer  duplicates  the  packet  and  forwards 
the  duplicate(s)  along  the  latter  tree  as  well.  This  lets  packets  reach  the  nodes  satisfying  a  faster 
by  taking  multiple  routes:  by  following  the  “gradient”  of  nodes  that  are  not  on  the  a  tree,  and 
via  the  trees  of  other  selected  atomic  propositions  that  intersect  the  a  tree.  (It  is  this  optimization 
that  requires  that  all  selected  atomic  propositions  be  carried  in  all  copies  of  the  packet.)  Figure  13 
illustrates  the  use  of  this  optimization,  as  well  as  the  use  of  the  pending  bits. 

Because  packets  can  encounter  a  selected  atomic  proposition’s  tree  multiple  times,  care  must 
be  taken  to  avoid  excessive  duplication  and  waste  of  bandwidth.  (For  example,  trees  that  intersect 


34 


in  two  places  would  result  in  packets  looping  forever.)  For  this  purpose  our  approach  requires 
a  duplicate  suppression  mechanism  that  keeps  packets  from  being  forwarded  over  the  same  link 
more  than  once  in  the  same  direction.  This  mechanism  can  be  implemented  in  several  ways.  One 
possibility  is  to  use  ephemeral  state  processing  (see  Section  5):  Each  packet  is  originated  with  a 
unique  tag;  a  packet  is  forwarded,  it  leaves  a  bit  of  ephemeral  state  on  each  interface  it  traverses; 
packets  carry  an  ESP  instruction  which  causes  a  packet  to  be  dropped  if  it  reaches  an  interface  where 
its  tag  is  already  stored.  Other  well-known  solutions  are  to  record  each  interface  traversed  in  the 
packet,  say  using  a  Bloom  filter,  or  to  have  packets  carry  a  hop-limit  field  that  is  decremented  each 
time  the  packet  is  forwarded,  and  drop  packets  when  the  hop-limit  field  reaches  zero. 

7.2.3  Evaluation  of  Layered  Approach 

To  evaluate  fhe  performance  and  overhead  cosfs  of  Ibis  implemenfafion  approach,  we  conducfed 
simulafion  experimenfs  on  fwo  versions.  The  firsl  is  fhe  basic  fwo-layer  service,  wifh  an  opfimiza- 
fion  fhaf  uses  addifional yiZter  slate  lo  keep  Irack  of  which  branches  of  fhe  free  for  one  alomic  propo- 
sifion  lead  lo  nodes  lhal  do  no!  (or  do)  salisfy  olher  alomic  proposilions.  The  second  version  did 
nol  use  filters,  bul  did  incorporate  supporl  for  hierarchical  atomic  propositions,  i.e.  allribules  lhal 
are  salisfied  only  by  nodes  salisfying  olher  allribules.  These  can  be  Ihoughl  of  as  address  prefixes: 
a  prefix  corresponds  lo  an  allribule  lhal  is  salisfied  by  nodes  whose  addresses  malch  Ihe  prefix.  The 
advanlage  of  slrucluring  Ihe  predicate  sel  in  Ibis  way  is  lhal  il  gives  Ihe  abilily  lo  refer  lo  groups 
of  nodes  wilh  higher-level  alomic  proposilions,  while  slill  relaining  Ihe  abilily  lo  specify  any  node 
individually.  We  call  Ihe  former  version  non-hierarchical  speccast,  and  Ihe  laller  hierarchical  spec- 
cast.  (Note  lhal  Ihe  lolal  number  of  alomic  proposilions  is  exponentially  larger  in  Ihe  hierarchical 
case.) 

To  measure  Ihe  performance  and  cosl  of  each  of  Ibis  speccasl  service,  we  carried  oul  Iwo  experi- 
menls.  The  firsl  experimenl  invesligales  Ihe  cosl  of  using  Ihe  various  services  lo  implemenl  unicast, 
Ihe  second  considers  Ihe  use  of  Ihe  services  for  multicast.  In  bolh  cases  we  measured  Ihe  nelwork 
delay,  nelwork  load,  and  nelwork  slate.  Eor  comparison,  in  addition  lo  our  Iwo  versions  of  spec¬ 
casl,  we  also  simulated  olher  routing  and  forwarding  approaches,  including:  a  generic  shorlesl  palh 
firsl  routing  algorilhm  (SPE)  lhal  simulates  Iradilional  unicasl  (and  IP  mullicasl)  routing;  a  generic 
publish-subscribe  service  based  on  a  broker  overlay,  in  which  a  packel  is  firsl  Iransmilled  lo  Ihe 
nearesl  broker,  which  forwards  il  lo  all  nodes  il  knows  aboul  lhal  satisfy  Ihe  destination  predicate; 
and  a  slruclured  (peer-lo-peer)  overlay  similar  lo  Tapeslry. 

In  Ihe  unicasl  experimenl,  each  node  in  Ihe  nelwork  was  assigned  a  unique  “address” — lhal  is, 
a  combination  of  allribules  satisfied  only  by  lhal  node.  How  Ihe  address  was  interpreted  depended 
on  Ihe  rouling  approach.  Bolh  Ihe  Broker  and  SPE  model  Ireal  addresses  as  “Hal”  (unslruclured) 
bils  slrings.  In  bolh  Ihe  Speccasl  services  and  Tapeslry,  Ihe  address  is  regarded  as  a  sequence  of 
base  b  digils.  Since  we  have  a  fixed  number  of  nodes  N,  Ihe  number  of  digils  needed  in  Ihe  address 
is  d  —  logi,N.  In  Ihe  Tapeslry  simulafion,  Ihe  digils  are  used  direclly  by  Ihe  routing  algorilhm  as 
described  above.  In  Ihe  non-hierarchical  speccasl  simulation,  we  associate  one  alomic  proposition 
wilh  each  (digil,  position)  pair.  Each  address  can  Ihus  be  uniquely  represented  as  a  conjunction  of  d 
alomic  propositions.  Thus  Ihe  number  of  alomic  propositions  in  Ihe  system  is  bxd.  To  measure  Ihe 
effecl  of  Ihe  base  b  on  Ihe  nelwork  slate,  load,  and  delay,  we  varied  b  from  2  lo  30  in  our  experiment. 

In  Ihe  hierarchical  speccasl  simulations,  Ihe  sel  of  alomic  propositions  conlains  a  single  alomic 
proposition  for  each  siring  of  k  digils,  for  k  belween  1  and  d.  These  alomic  propositions  are  ar- 
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ranged  hierarchically  in  essentially  the  prefix-based  structure  described  earlier.  So  the  attribute 
corresponding  to  the  string  1223  has  a  “parent”  attribute  corresponding  to  the  string  122. 

Before  presenting  the  results,  we  must  explain  one  other  concept.  Locality  is  a  correlation 
between  a  node’s  position  in  the  topology  and  the  set  of  predicates  satisfied  by  a  node.  With  locality, 
knowing  that  node  n  satisfies  atomic  proposition  a  removes  some  uncertainty  about  the  probability 
that  other  nodes  near  n  in  the  topology  will  also  satisfy  o. 

To  evaluate  the  affect  of  locality  on  performance  and  overhead,  we  ran  two  versions  of  each 
experiment  using  transit-stub  graph  models  generated  with  the  well-known  GT-ITM  topology  mod¬ 
eling  package.  In  the  first  test,  we  randomly  assigned  addresses  to  nodes.  As  a  result,  the  probability 
of  nodes  with  “similar”  addresses  being  near  one  another  was  relatively  low  (i.e.,  no  locality).  In 
the  second  test,  addresses  were  assigned  according  to  node  numbers  in  the  GT-ITM  graph  structure. 
Because  of  the  way  transit-stub  graphs  are  constructed,  nodes  near  each  other  in  the  topology  have 
similar  node  numbers,  and  so  received  similar  addresses  in  the  second  test.  As  a  result,  nodes  in 
the  same  network  domain  usually  have  a  common  address  prefix.  Each  topology  consisted  of  3080 
nodes  comprising  four  transit  domains  averaging  14  nodes  each,  with  each  transit  node  connecting 
to  3  stub  domains  averaging  18  nodes  each.  Each  graph  also  contained  12  extra  transit-stub  edges 
and  56  additional  stub-stub  edges;  these  extra  edges  increase  the  number  of  paths  between  stub 
nodes  in  the  graph,  and  thus  provide  more  diversity  of  routing. 

We  also  simulated  the  use  of  speccast  to  provide  multicast  service  in  a  straightforward  way  using 
atomic  propositions  in  place  of  multicast  addresses.  To  send  a  packet  to  the  group,  a  sender  simply 
addresses  the  packet  to  the  group’s  atomic  proposition.  Unlike  conventional  multicast,  our  multicast 
service  is  more  expressive  and  could  also  be  used  to  send  packets  to  the  unions  or  intersections  of 
groups.  Using  hierarchical  speccast  service,  groups  could  also  be  partitioned  into  subgroups,  which 
might  be  useful,  say,  for  subcasting. 
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Eigure  14:  Normalized  network  delay  for  unicast  delivery. 

The  graphs  in  Eigures  14-16  plot  the  metrics  discussed  earlier  as  a  function  of  the  “base”  of 
the  (unicast)  addresses.  Eigure  14  shows  the  network  delay  stretch,  computed  as  the  ratio  of  the 
measured  delay  to  the  optimal  (shortest  path)  delay,  when  addressed  are  assigned  with  and  without 
locality.  Eigure  15  shows  the  network  load  (measured  in  number  of  packet-hops)  imposed  on  the 
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Figure  16:  Network  state  required  for  unieast  delivery. 
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system  by  each  algorithm.  Unlike  delay,  the  network  load  is  noticeably  affected  by  locality.  Finally, 
Figure  16  shows  the  amount  of  state  each  algorithm  requires  in  the  network.  Although  speccast 
competes  fairly  well  in  terms  on  delay  and  network  load,  its  greater  flexibility  comes  at  the  cost  of 
additional  forwarding  state  kept  in  the  network.  Without  locality,  each  tree  contains  many  nodes 
that  do  not  satisfy  the  atomic  proposition,  but  yet  must  maintain  the  state.  With  locality,  the  number 
of  such  nodes  drops  significantly,  as  does  the  network  state. 

Figure  17  shows  the  network  delay  and  network  load  when  speccast  is  used  for  multicast  de¬ 
livery.  The  results  confirm  our  expectations:  Because  speccast  constructs  a  shared  distribution  tree. 


Number  of  group  members 


Figure  17:  Normalized  delay  and  network  load  for  delivery  to  groups  of  receivers. 

it  performs  much  like  a  core -based  multicast  tree,  reducing  the  network  load  (even  below  1)  at  the 
expense  of  longer  delays  (stretch  >  1). 

7.3  A  Link-State  Implementation 

In  addition  to  the  distance-vector  approach  described  above,  we  have  implemented  speccast  using  a 
different  approach  based  on  a  link-state  routing  protocol  and  source-routing.  The  advantages  of  this 
approach  are  that  (i)  it  requires  very  little  knowledge  about  the  language  used  to  specify  destination 
predicates;  (ii)  it  does  not  assume  or  require  the  existence  of  any  globally-assigned  node  identifiers; 
(iii)  it  can  support  the  aggregation  of  node  predicates  within  areas,  making  a  subgraph  of  the  network 
look  like  a  single  node,  and  thus  can  in  principle  support  large-scale  networks  (though  we  have  not 
implemented  this  capability);  and  (iv)  it  admits  the  possibility  of  representing  destination  predicates 
as  simple  programs  in  packets. 

Our  implementation,  which  is  written  in  Java,  comprises  code  (i.e.  a  seperate  class)  for  each 
of  the  routing  and  forwarding  components,  as  well  as  a  Hello  protocol.  The  Hello  protocol  en¬ 
ables  a  node  to  locate  its  neighbors  in  the  network,  and  assigns  (random)  identifiers  to  each  link. 
The  routing  algorithm  performs  two  functions.  First,  each  node  periodically  transmits  local  routing 
information — in  the  form  of  a  link-state  announcement,  or  LSA — to  its  neighbors.  Each  ESA  con¬ 
tains  the  node  predicate  for  the  originating  node,  and  a  list  of  links  to  which  it  is  connected.  Second, 
information  from  incoming  ESAs  is  used  to  construct  a  network  map,  including  each  node  and  the 
predicate  it  satisfies,  as  well  as  the  links  connecting  nodes  to  each  other. 

Packets  are  forwarded  using  a  source  routing  approach.  To  originate  a  packet,  a  node  consults 
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its  network  map  to  determine  whieh  node(s)  satisfy  the  destination  predieate.  It  then  determines  a 
(branehing)  path  to  be  followed  by  the  paeket  to  all  sueh  nodes.  This  tree  is  eneoded  in  the  paeket 
header.  When  a  paeket  arrives  at  a  node,  the  destination  predieate  is  eheeked  to  see  whether  the  node 
satisfies  it;  if  so  it  is  delivered  to  the  higher  level.  Then  the  paeket  is  forwarded  on  the  interfaees 
indieated  by  the  tree  eontained  in  the  paeket. 

Our  implementation  uses  the  simple  predieate  language  deseribed  in  the  previous  subseetion, 
i.e.  both  node  and  destination  predieates  are  in  the  form  of  disjunetions  of  eonjunetions  of  at¬ 
tributes.  A  paeket  is  delivered  to  a  node  if  (and  only  if)  it  is  possible  that  the  node  satisfies 
fhe  desfinafion  predieafe,  i.e.  if  af  leasf  one  eonjunefion  in  one  of  fhe  predieafes  eonfains  all  af- 
fribufes  of  some  eonjunefion  of  fhe  ofher.  Affribufe  names  are  eneoded  as  ease-sensifive  sfrings 
of  leffers  and  numbers.  Conjunefion  and  disjunefion  are  indieafed  by  &  and  | ,  respeefively  (e.g. 
”abc&def  |  qwe  &  asd”). 

The  profoeol  is  implemenfed  as  a  Java  Class  (“Node”),  whieh  ineludes  an  API  wifh  send  { ) , 
setPredicate  { )  and  registerReceiver  { )  mefhods.  (The  reeeiver  inferfaee  musf  be  im- 
plemenfed  by  fhe  higher-level  profoeol.)  An  insfanee  of  fhe  node  elass,  when  sfarfed,  broadeasfs 
messages  fo  ifs  neighbors  on  all  eonfigured  interfaees,  and  is  fhen  ready  fo  forward  paekefs.  Our 
eode  eonfains  several  sample  applieafions:  NetTest,  (fo  fesf  wifh  a  nefwork  fopology  ereafed  by 
GT-ITM)  and  a  simple  3  node  fopology  fesf. 

8  Other  Technologies 

In  addifion  fo  fhe  primary  feehnologies  deseribed  above,  fhe  AefiveCasf  projeef  has  produeed  ofher 
feehnologies,  whieh  we  deseribe  nexf. 

8.1  FPAC:  Fast,  Fixed-cost  Packet  Authentication 

One  of  fhe  problems  fhaf  arises  in  providing  enhaneed  serviees  like  example  eoneasf,  or  any  ofher 
serviee  in  whieh  some  paekefs  reeeive  serviee  beyond  fhe  normal  besl-efforl  forwarding,  is  fhaf  of 
ensuring  fhaf  only  paekefs  fhaf  are  enfifled  fo  fhe  serviee  reeeive  if.  In  parfieular,  if  resourees  (e.g. 
bandwidfh)  are  reserved  for  one  user’s  paekefs,  and  some  ofher  (malieious)  user  sends  “spoofed” 
paekefs,  impersonafing  fhe  firsf  user,  fhe  spoofed  paekefs  may  use  up  fhe  reserved  resourees,  fhus 
denying  serviee  fo  fhe  legifimafe  user.  The  usual  approaeh  fo  defeefing  spoofed  paekefs  is  fo  use 
a  erypfographie  message  aufhenfieafion  eheek  (MAC)  eode.  However,  any  per-paekef  eheek  musf 
be  (i)  eapable  of  being  performed  af  wire  speed  for  minimum-sized  paekefs,  and  (ii)  immune  fo 
replay  affaeks.  If  paekefs  eannof  be  proeessed  af  leasf  as  fasf  as  fhey  arrive,  fhen  fhe  eheek  ifself  is 
vulnerable  fo  a  denial-of-serviee  affaek. 

We  Iherefore  developed  and  implemenfed  a  lighfweighf  aufhenfieafion  eode  ealled  FPAC,  whieh 
ean  be  implemenfed  for  a  fixed  per-paekef  eosf.  FPAC,  whieh  is  deseribed  in  a  paper  presented  af 
fhe  INFOCOM  2002  eonferenee  [11],  is  designed  speeifieally  fo  profeef  againsf  unaufhorized  aeeess 
fo  resourees  fhaf  are  eonsumed  in  an  amounf  proporfional  fo  a  paekefs  lengfh.  The  eosf  is  fixed 
because  only  a  fixed-size  porfion  of  fhe  packef  (i.e.  a  “shim”  header  befween  fhe  IP  header  and 
fhe  fransporf  header)  is  acfually  checked  for  aulhenficify.  The  advanfage  of  fhis  approach  is  fhaf  if 
can  be  implemenfed  wifhouf  requiring  fhe  enfire  packef.  The  fradeoff  is  fhaf  FPAC  eannof  be  used 
for  end-fo-end  infegrify  checks;  however,  fhere  are  well-known  good  solufions  fo  fhe  problem  of 
end-fo-end  aulhenficify /infegrify  verificalion. 
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FPAC  uses  a  symmetric-key  cipher  to  hash  a  shared  secret.  It  requires  that  the  shared  secret 
be  distributed  to  all  nodes  that  have  reserved  resources  for  a  flow.  We  did  not  develop  a  specific 
solution  for  the  key-distribution  problem,  although  we  believe  the  secure  concast  signaling  protocol 
could  be  used  for  the  purpose. 

8.2  A  Generic  Grouping  Service 

We  have  also  studied  the  design  and  implementation  of  a  generic,  distributed  grouping  service.  The 
service  is  intended  to  help  large-scale  applications  distribute  participants  among  groups.  In  recent 
years,  the  design  and  use  of  various  forms  of  overlay  networks  has  been  a  topic  of  interest  in  the  net¬ 
working  field.  Overlay  networks  iterate  the  “catenet”  idea  that  led  to  the  original  Internet,  by  treating 
the  Internet  itself  as  a  lower-level  network  and  building  a  higher-level  topology  using  Internet-level 
“virtual  links”.  Such  topologies,  however,  can  be  very  inefficient  if  they  are  constructed  in  such  a 
way  that  neighbor  relationships  at  the  overlay  level  do  not  respect  the  topology  of  the  underlying 
network.  Our  grouping  service  is  intended  to  help  with  this  problem — and  related  problems,  such 
as  determining  which  multicast  receivers  should  use  which  recovery  server — ^by  taking  application- 
level  information  from  each  participant,  combining  it  with  network-level  information,  and  returning 
an  indication  of  which  participants  should  be  in  the  same  groups.  More  precisely,  the  service  allows 
the  application  to  define  a  “distance”  metric  on  application  information,  and  combines  that  with  a 
network-level  metric,  in  such  a  way  that  some  combination  (also  specified  by  the  application)  of  the 
two  metrics  is  minimized. 

The  grouping  services  we  have  studied  so  far  are  designed  to  leverage  advanced/programmable 
network  services.  Along  with  the  design  of  a  general  “grouping  API”,  we  have  published  descrip¬ 
tions  of  both  concast-based  [14]  and  ESP-based  [17]  grouping  service  implementations. 

9  Summary  of  Work  Products 

In  addition  to  the  technologies  and  prototype  implementations^  already  described  in  this  report,  this 
project  has  produced  the  following  outputs: 

•  20  refereed  or  invited  publications,  which  have  appeared  in  top  venues  for  networking  re¬ 
search.  All  publications  which  have  appeared  to  date  are  listed  below  under  the  heading 
“References”;  at  least  two  more  are  currently  in  preparation  for  submission. 

•  One  patent  application  was  filed,  for  “System  and  Process  for  Providing  Auxiliary  Informa¬ 
tion  for  a  Packet-Switched  Network  of  Shared  Nodes  Using  Dedicated  Associative  Store”, 
inventors  Kenneth  L.  Calvert  and  James  N.  Griffioen. 

•  Work  done  as  part  of  this  project  has  made  up  or  will  appear  in  the  PhD  thesis  of  at  least  four 
students:  Su  Wen,  Amit  Sehgal,  and  Leonid  Poutievski  of  the  University  of  Kentucky,  and 
Youngsu  Chae  of  the  Georgia  Institute  of  Technology. 

•  Work  done  on  various  aspects  of  this  project  formed  the  Masters  Degree  Project  for  two 
students,  Srini  Venkatraman  and  Chetan  Singh  Dillon  of  the  University  of  Kentucky. 

Some  of  the  text  of  this  report  was  contributed  by  coauthors  of  papers  in  the  list  below. 

^All  of  which  are  available  at  the  project  web  site  (http  :  //protocols  .  netlab .  uky  .  edu/~acast/) 
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